Oct 31 12:24:12.730381 ip-10-0-142-45 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 12:24:12.730396 ip-10-0-142-45 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 12:24:12.730405 ip-10-0-142-45 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 12:24:12.730746 ip-10-0-142-45 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 12:24:22.925907 ip-10-0-142-45 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 12:24:22.925922 ip-10-0-142-45 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 53aae745adf14ced92d1b9b46ae701f1 -- Oct 31 12:26:56.388737 ip-10-0-142-45 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:26:56.756119 ip-10-0-142-45 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:26:56.756119 ip-10-0-142-45 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:26:56.756119 ip-10-0-142-45 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:26:56.756119 ip-10-0-142-45 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:26:56.762432 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762369 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.762432 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762431 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.762432 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762435 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762438 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762441 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762445 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762449 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762454 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762457 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762474 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762480 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762483 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762491 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762494 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762497 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762500 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762502 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762505 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762507 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762510 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.762539 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762512 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762515 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762518 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762520 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762523 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762525 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762528 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762530 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762533 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762536 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762538 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762541 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762543 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762547 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762550 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762553 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762556 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762558 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762561 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762563 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.762971 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762566 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762569 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762571 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762573 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762576 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762579 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762581 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762584 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762586 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762589 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762591 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762594 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762597 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762599 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762604 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762607 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762609 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762612 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762615 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762617 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.763435 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762620 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762622 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762625 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762627 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762630 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762632 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762635 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762639 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762642 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762645 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762648 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762650 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762653 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762655 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762659 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762662 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762665 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762667 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762670 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762673 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.763916 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762675 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762678 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762680 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762683 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762685 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762691 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762694 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762697 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762699 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762702 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762704 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762707 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762710 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762712 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762715 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762717 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762720 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762722 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.764392 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762725 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762728 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762730 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762733 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762735 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.762737 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763495 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763502 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763505 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763508 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763510 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763513 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763517 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763520 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763523 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763527 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763531 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763534 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763537 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763541 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.764849 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763544 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763546 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763549 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763552 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763554 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763557 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763560 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763563 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763566 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763569 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763571 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763574 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763579 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763582 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763584 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763587 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763589 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763592 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.765329 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763594 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763597 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763599 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763602 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763604 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763607 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763610 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763612 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763615 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763618 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763620 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763623 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763626 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763629 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763631 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763634 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763636 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763639 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763642 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763645 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.765794 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763648 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763650 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763653 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763655 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763658 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763661 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763663 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763666 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763668 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763671 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763673 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763676 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763678 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763681 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763684 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763686 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763689 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763691 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763694 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.766272 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763697 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763700 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763703 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763706 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763709 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763712 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763715 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763717 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763720 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763722 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763725 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763727 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763730 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763734 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763738 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763741 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763744 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763746 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763749 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.766741 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763752 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763754 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763757 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763759 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763762 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763764 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763767 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763769 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763772 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763775 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763778 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763782 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763784 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763787 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.763790 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764248 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764253 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764256 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764258 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764261 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.767191 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764264 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764267 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764270 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764272 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764275 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764278 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764280 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764283 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764285 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764288 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764292 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764295 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764298 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764300 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764303 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764305 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764308 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764310 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764313 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.767665 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764316 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764318 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764321 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764324 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764327 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764329 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764332 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764335 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764337 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764340 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764342 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764344 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764347 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764350 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764352 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764355 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764357 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764360 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764362 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764365 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.768106 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764368 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764370 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764372 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764375 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764378 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764380 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764383 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764386 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764389 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764392 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764394 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764397 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764399 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764402 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764405 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764407 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764410 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764412 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764415 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.768607 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764417 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764420 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764423 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764425 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764428 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764432 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764435 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764439 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764441 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764444 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764446 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764449 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764451 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764454 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764456 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764473 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764478 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764481 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764485 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.769059 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764490 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764493 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764496 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764498 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764501 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764503 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764506 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764508 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764511 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764514 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764517 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764519 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764522 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764527 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764530 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764533 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764535 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764538 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764542 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.769527 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764545 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764548 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764550 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.764553 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765103 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765110 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765116 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765121 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765126 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765129 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765133 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765137 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765141 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765144 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765147 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765150 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765153 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765156 2578 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765159 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765162 2578 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765167 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765169 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765174 2578 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765177 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:26:56.770038 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765180 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765183 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765187 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765190 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765194 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765197 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765201 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765205 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765208 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765211 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765214 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765217 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765221 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765224 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765227 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765230 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765233 2578 flags.go:64] FLAG: --enable-server="true" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765236 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765240 2578 flags.go:64] FLAG: --event-burst="100" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765243 2578 flags.go:64] FLAG: --event-qps="50" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765246 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765249 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765252 2578 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765256 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:26:56.770683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765259 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765262 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765265 2578 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765268 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765272 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765275 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765278 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765281 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765284 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765287 2578 flags.go:64] FLAG: --feature-gates="" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765291 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765294 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765298 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765301 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765304 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765308 2578 flags.go:64] FLAG: --help="false" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765311 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765315 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765318 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765321 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765324 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765327 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765330 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:26:56.771297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765334 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765337 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765340 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765343 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765346 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765349 2578 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765352 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765354 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765358 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765360 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765363 2578 flags.go:64] FLAG: --lock-file="" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765366 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765369 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765375 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765381 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765384 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765387 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765389 2578 flags.go:64] FLAG: --logging-format="text" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765392 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765396 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765399 2578 flags.go:64] FLAG: --manifest-url="" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765402 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765406 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765409 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765414 2578 flags.go:64] FLAG: --max-pods="110" Oct 31 12:26:56.771987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765417 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765420 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765423 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765426 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765429 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765432 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765435 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765443 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765446 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765449 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765452 2578 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765455 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765457 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765474 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765477 2578 flags.go:64] FLAG: --port="10250" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765480 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765483 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0107acac4cee1cdd1" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765486 2578 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765489 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765492 2578 flags.go:64] FLAG: --register-node="true" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765496 2578 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765503 2578 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765506 2578 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765509 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765512 2578 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765516 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:26:56.772847 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765519 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765522 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765525 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765528 2578 flags.go:64] FLAG: --runonce="false" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765531 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765537 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765540 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765543 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765546 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765549 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765552 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765555 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765558 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765561 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765564 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765567 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765570 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765573 2578 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765576 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765582 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765584 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765587 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765591 2578 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765594 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765597 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:26:56.773492 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765600 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765604 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765607 2578 flags.go:64] FLAG: --v="2" Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765612 2578 flags.go:64] FLAG: --version="false" Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765616 2578 flags.go:64] FLAG: --vmodule="" Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765619 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.765622 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765752 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765755 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765758 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765761 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765764 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765768 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765771 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765774 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765776 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765780 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765782 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765785 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765787 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765790 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.774081 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765793 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765796 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765798 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765801 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765804 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765807 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765809 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765812 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765815 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765817 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765820 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765823 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765827 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765830 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765840 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765843 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765846 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765849 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765852 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765854 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.774655 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765857 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765860 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765862 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765868 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765871 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765874 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765877 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765879 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765882 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765884 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765887 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765890 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765892 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765895 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765897 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765900 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765902 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765905 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.775113 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765907 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765909 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765912 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765915 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765917 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765921 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765924 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765926 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765929 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765931 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765934 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765936 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765939 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765941 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765944 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765946 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765949 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765953 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765957 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.775602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765961 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765964 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765967 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765970 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765972 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765975 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765977 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765980 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765982 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765985 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765987 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765990 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765992 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765995 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765997 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.765999 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766003 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766007 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766011 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.776048 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766014 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766016 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766019 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766021 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766024 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766026 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766029 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766031 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766034 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766037 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766040 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766042 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766045 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766047 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766050 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.766076 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:26:56.776525 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766111 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766115 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766118 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766120 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766123 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766125 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766128 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766131 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766133 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766136 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766140 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766147 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766151 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766155 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766158 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766162 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766165 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766167 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766170 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.776901 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766173 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766175 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766178 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766180 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766183 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766185 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766188 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766191 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766193 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766197 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766201 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766204 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766213 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766215 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766218 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766221 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766223 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766226 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766228 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766231 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.777354 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766233 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766236 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766238 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766240 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766243 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766245 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766248 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766251 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766256 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766259 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766261 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766264 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766267 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766269 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766272 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766274 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766278 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766283 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766285 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766288 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.777866 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766291 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766294 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766297 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766299 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766302 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766305 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766307 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766310 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766313 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766315 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766318 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766320 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766323 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766325 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766328 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766331 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766333 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766336 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766338 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766341 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.778331 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766344 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766347 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766350 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766352 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766355 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766357 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766360 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766362 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766365 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766369 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766371 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766374 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766376 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766379 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766381 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766384 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766386 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766389 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766392 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.778805 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766395 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766397 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766400 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766402 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766405 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766407 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766410 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.766413 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.773444 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.773478 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.774168 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.774175 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774238 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774243 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774247 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774250 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774253 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774256 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.779249 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774261 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774264 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774267 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774270 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774273 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774275 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774278 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774281 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774283 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774286 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774288 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774291 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774293 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774297 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774299 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774302 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774305 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774308 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774311 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774313 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.779709 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774316 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774319 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774322 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774325 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774327 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774330 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774332 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774335 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774338 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774343 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774345 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774348 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774350 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774353 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774355 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774358 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774361 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774363 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774373 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.780179 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774376 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774379 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774381 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774384 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774387 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774389 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774392 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774394 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774397 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774400 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774402 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774405 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774407 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774410 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774413 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774416 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774418 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774421 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774424 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.780705 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774426 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774429 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774432 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774434 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774437 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774440 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774442 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774444 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774447 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774449 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774452 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774457 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774482 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774486 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774489 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774492 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774494 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774497 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774499 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.781152 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774502 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774505 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774507 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774509 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774512 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774516 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774521 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774526 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774530 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774533 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774536 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774538 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774541 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774544 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774546 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774549 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774552 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774555 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774557 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.781667 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774560 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.774588 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774730 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774735 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774738 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774741 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774744 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774752 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774755 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774758 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774760 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774763 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774765 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774768 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774771 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774775 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.782232 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774778 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774782 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774784 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774787 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774790 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774793 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774795 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774798 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774800 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774803 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774805 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774808 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774810 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774813 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774815 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774818 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774821 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774823 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774826 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.782646 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774829 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774831 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774834 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774836 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774839 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774847 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774850 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774853 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774855 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774858 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774860 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774863 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774865 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774868 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774871 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774875 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774878 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774881 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774884 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.783101 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774886 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774889 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774891 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774894 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774896 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774899 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774901 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774904 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774906 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774909 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774911 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774914 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774917 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774919 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774922 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774924 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774927 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774929 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774932 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774940 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.783685 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774943 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774945 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774948 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774950 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774953 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774955 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774958 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774960 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774963 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774966 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774968 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774970 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774973 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774976 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774978 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774981 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774983 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774986 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774988 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774991 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.784175 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774993 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774996 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.774999 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775001 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775004 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775006 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775009 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775011 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775014 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775016 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775019 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775021 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775029 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775032 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.775055 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775092 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:26:56.784672 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775096 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775099 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775102 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775105 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775108 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775111 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775113 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775117 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775122 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775125 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775128 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775131 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775133 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775136 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775138 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775141 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775144 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775146 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775148 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:26:56.785065 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775151 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775154 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775157 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775159 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775162 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775165 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775168 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775170 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775172 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775175 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775184 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775186 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775189 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775191 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775194 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775196 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775199 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775202 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775205 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:26:56.785580 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775207 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775210 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775213 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775215 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775219 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775222 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775225 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775229 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775231 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775234 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775236 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775239 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775241 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775244 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775247 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775249 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775252 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775254 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775257 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775259 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:26:56.786071 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775262 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775264 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775269 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775272 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775280 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775283 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775285 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775288 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775290 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775293 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775295 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775298 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775300 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775303 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775305 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775308 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775310 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775313 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775315 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775318 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:26:56.786602 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775320 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775323 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775325 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775328 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775330 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775333 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775335 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775338 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775340 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775343 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775345 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775349 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775351 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775354 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775357 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775360 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775362 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775371 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775373 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:26:56.787124 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775376 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775378 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775381 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775383 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775386 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775388 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775391 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:56.775393 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.775484 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.778174 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.779027 2578 server.go:1014] "Starting client certificate rotation" Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.779124 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.779158 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.780919 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:26:56.787656 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.782000 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:26:56.791096 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.791056 2578 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:26:56.794098 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.794080 2578 log.go:25] "Validated CRI v1 image API" Oct 31 12:26:56.794942 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.794919 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:26:56.797019 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.796994 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80b4c50b-55a8-4f1b-a3e1-91c614682cb8:/dev/nvme0n1p4 842b1835-9a5d-4357-9e49-bd18f54c6546:/dev/nvme0n1p3] Oct 31 12:26:56.797081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.797018 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 12:26:56.802605 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.802491 2578 manager.go:217] Machine: {Timestamp:2026-10-31 12:26:56.801141026 +0000 UTC m=+0.304440045 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096579 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c75ab32dc0212b543d01a00ad297 SystemUUID:ec21c75a-b32d-c021-2b54-3d01a00ad297 BootID:53aae745-adf1-4ced-92d1-b9b46ae701f1 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:a3:4f:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:a3:4f:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:5d:4b:f1:c0:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:26:56.802605 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.802605 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:26:56.802713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.802669 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:26:56.803728 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.803704 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:26:56.803862 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.803729 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-45.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:26:56.804400 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.804391 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:26:56.804432 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.804403 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:26:56.804432 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.804425 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:26:56.804495 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.804436 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:26:56.805246 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.805236 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:26:56.809008 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.808996 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:26:56.809840 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.809801 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:26:56.810692 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.810630 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:26:56.810768 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.810722 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:26:56.810768 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.810750 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:26:56.810955 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.810942 2578 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:26:56.810987 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.810962 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:26:56.811293 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.811278 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:26:56.813347 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.813327 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:26:56.813924 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.813907 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:26:56.818033 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818018 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:26:56.818716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818702 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:26:56.818762 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818735 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:26:56.818892 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818882 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:26:56.818924 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818898 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:26:56.818924 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818904 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:26:56.818924 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818910 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:26:56.818924 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818915 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:26:56.818924 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.818922 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:26:56.819481 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.819457 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:26:56.819509 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.819482 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:26:56.819509 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.819490 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:26:56.819509 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.819495 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:26:56.819509 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.819504 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:26:56.819632 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.819512 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:26:56.820747 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.820737 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:26:56.820779 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.820751 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:26:56.824156 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.824141 2578 server.go:1263] "Started kubelet" Oct 31 12:26:56.824270 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.824223 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:26:56.824338 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.824251 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:26:56.824338 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.824314 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:26:56.824816 ip-10-0-142-45 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:26:56.825328 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.825314 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:26:56.826322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.826302 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:26:56.828033 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.827643 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-45.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 12:26:56.830310 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.830287 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 12:26:56.830426 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.830412 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:26:56.831015 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.831001 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:26:56.831780 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.831753 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:26:56.831780 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.831783 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:26:56.831885 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.831857 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:26:56.831920 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.831902 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:26:56.831920 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.831908 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:26:56.832004 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.831958 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:56.832219 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.832200 2578 factory.go:153] Registering CRI-O factory Oct 31 12:26:56.832292 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.832224 2578 factory.go:223] Registration of the crio container factory successfully Oct 31 12:26:56.832292 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.832289 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:26:56.832358 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.832299 2578 factory.go:55] Registering systemd factory Oct 31 12:26:56.832358 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.832322 2578 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:26:56.832358 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.832338 2578 factory.go:103] Registering Raw factory Oct 31 12:26:56.832358 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.832353 2578 manager.go:1199] Started watching for new ooms in manager Oct 31 12:26:56.833213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.833194 2578 manager.go:319] Starting recovery of all containers Oct 31 12:26:56.840409 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.840385 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-45.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 12:26:56.840798 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.840777 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-45.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 12:26:56.841806 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.840958 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-258gc" Oct 31 12:26:56.841957 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.840833 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-45.us-east-2.compute.internal.18e39ceca55c9640 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-45.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-45.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:26:56.824112704 +0000 UTC m=+0.327411722,LastTimestamp:2026-10-31 12:26:56.824112704 +0000 UTC m=+0.327411722,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-45.us-east-2.compute.internal,}" Oct 31 12:26:56.842961 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.842921 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 12:26:56.843052 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.843033 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 12:26:56.845550 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.845535 2578 manager.go:324] Recovery completed Oct 31 12:26:56.850808 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.850795 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:56.851300 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.851284 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-258gc" Oct 31 12:26:56.853363 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.853345 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:56.853434 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.853374 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:56.853434 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.853389 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:56.854638 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.854623 2578 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:26:56.854638 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.854637 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:26:56.854772 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.854653 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:26:56.854772 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.854630 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:56.855330 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.855314 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:56.855407 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.855383 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:56.855407 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.855401 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:56.855696 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.855604 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-45.us-east-2.compute.internal.18e39ceca71ae799 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-45.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-45.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-45.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:26:56.853362585 +0000 UTC m=+0.356661606,LastTimestamp:2026-10-31 12:26:56.853362585 +0000 UTC m=+0.356661606,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-45.us-east-2.compute.internal,}" Oct 31 12:26:56.857102 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.857089 2578 policy_none.go:50] "Start" Oct 31 12:26:56.857102 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.857103 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:26:56.857208 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.857115 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:26:56.858144 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.858081 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-45.us-east-2.compute.internal.18e39ceca71b316f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-45.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-45.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-45.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:26:56.853381487 +0000 UTC m=+0.356680511,LastTimestamp:2026-10-31 12:26:56.853381487 +0000 UTC m=+0.356680511,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-45.us-east-2.compute.internal,}" Oct 31 12:26:56.859146 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.859135 2578 policy_none.go:44] "Start" Oct 31 12:26:56.894106 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.894070 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:26:56.896729 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.896715 2578 manager.go:341] "Starting Device Plugin manager" Oct 31 12:26:56.896791 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.896747 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:26:56.896791 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.896756 2578 server.go:87] "Starting device plugin registration server" Oct 31 12:26:56.897021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.897008 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:26:56.897086 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.897023 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:26:56.897622 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.897493 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:26:56.897622 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.897566 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:26:56.897622 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.897573 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:26:56.897793 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.897721 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:26:56.897793 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.897767 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:56.962002 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.961972 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:26:56.963186 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.963165 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:26:56.963273 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.963191 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:26:56.963273 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.963214 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:26:56.963273 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:56.963255 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 12:26:56.966702 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.966684 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:26:56.997818 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.997797 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:56.998618 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.998605 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:56.998693 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.998633 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:56.998693 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.998646 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:56.998693 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:56.998669 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.008472 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.008420 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.008472 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.008444 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-45.us-east-2.compute.internal\": node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.053872 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.053729 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.063761 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.063733 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal"] Oct 31 12:26:57.063816 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.063797 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:57.064528 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.064514 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:57.064599 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.064540 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:57.064599 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.064555 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:57.066688 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.066676 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:57.066866 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.066852 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.066916 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.066881 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:57.067325 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.067311 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:57.067395 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.067339 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:57.067395 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.067348 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:57.067395 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.067311 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:57.067395 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.067383 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:57.067395 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.067394 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:57.069419 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.069403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.069519 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.069430 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:57.070091 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.070076 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:57.070091 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.070102 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:57.070208 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.070112 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:57.103026 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.102996 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" node="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.105353 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.105336 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" node="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.133676 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.133655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.133751 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.133680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9561195d04d47702d33d861addb8e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"2e9561195d04d47702d33d861addb8e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.133751 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.133698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.154787 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.154765 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.233953 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.233931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.234044 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.233962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.234044 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.233980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9561195d04d47702d33d861addb8e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"2e9561195d04d47702d33d861addb8e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.234044 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.234032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9561195d04d47702d33d861addb8e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"2e9561195d04d47702d33d861addb8e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.234044 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.234033 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.234186 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.234043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.255546 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.255517 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.356219 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.356166 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.406354 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.406332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.408530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.408513 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:57.457013 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.456991 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.557445 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.557420 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.658012 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.657961 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.758452 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.758429 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.778614 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.778582 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 12:26:57.778754 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.778734 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:26:57.856783 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.856734 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:21:57 +0000 UTC" deadline="2028-06-11 01:20:34.405584199 +0000 UTC" Oct 31 12:26:57.856783 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:57.856773 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14124h53m36.548813592s" Oct 31 12:26:57.858895 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.858877 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:57.958973 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:57.958916 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:58.025283 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.025265 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:26:58.026364 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.026347 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:26:58.026459 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.026379 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:26:58.026459 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.026391 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:26:58.060063 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:58.060040 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:58.160222 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:58.160185 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:26:58.197839 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:58.197802 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86c20a2519affed9d4fc6daf0172b42.slice/crio-c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073 WatchSource:0}: Error finding container c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073: Status 404 returned error can't find the container with id c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073 Oct 31 12:26:58.202824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.202808 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:26:58.207248 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:58.207225 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9561195d04d47702d33d861addb8e7.slice/crio-bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091 WatchSource:0}: Error finding container bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091: Status 404 returned error can't find the container with id bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091 Oct 31 12:26:58.228439 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.228416 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:26:58.234491 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.234457 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:26:58.256241 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.256220 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:26:58.332215 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.332188 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:58.348778 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.348759 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:26:58.349880 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.349867 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:26:58.359777 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.359762 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:26:58.669934 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.669881 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:26:58.812933 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.812901 2578 apiserver.go:52] "Watching apiserver" Oct 31 12:26:58.820849 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.820825 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:26:58.821204 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.821177 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wnnvc","openshift-ovn-kubernetes/ovnkube-node-jq9k9","kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-kr4dj","openshift-network-diagnostics/network-check-target-zcknp","openshift-multus/multus-p2ppd","openshift-network-operator/iptables-alerter-s2wpv","kube-system/global-pull-secret-syncer-g7zbh","openshift-cluster-node-tuning-operator/tuned-sv5jl","openshift-dns/node-resolver-vjlsm","openshift-image-registry/node-ca-dz7qv","kube-system/konnectivity-agent-6b687","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh"] Oct 31 12:26:58.823840 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.823823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.826565 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.826082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:58.828292 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.828271 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:26:58.829769 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.828666 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ljk6\"" Oct 31 12:26:58.829769 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.828676 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:26:58.829769 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.828689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.830105 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.830087 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:26:58.830182 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.830150 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:26:58.830306 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.830291 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bbb6f\"" Oct 31 12:26:58.831088 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.831064 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 12:26:58.831486 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.831466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.832411 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.832395 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:26:58.832529 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.832461 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:26:58.832529 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.832503 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x5rvd\"" Oct 31 12:26:58.832640 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.832584 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:26:58.833786 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.833757 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:58.833875 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:58.833851 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:26:58.835281 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.835260 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:26:58.835388 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.835351 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:26:58.835548 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.835526 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:26:58.835683 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.835629 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ng46x\"" Oct 31 12:26:58.836065 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.836047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:26:58.836301 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:58.836274 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:26:58.838369 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.838351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:58.838509 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:58.838411 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.842783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-modprobe-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.842823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-run\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.842840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.842857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsz9\" (UniqueName: \"kubernetes.io/projected/958a8aee-281c-45d9-8c87-ee2ce835fbee-kube-api-access-qcsz9\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.842935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-device-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.842983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-registration-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-sys-fs\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-kubernetes\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-host\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843213 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-tuned\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32afbb71-9024-4d12-a27a-994892cf38e4-konnectivity-ca\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-systemd\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-socket-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ecc097-1c88-4f8c-928c-708b9a628a72-host-slash\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-conf\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32afbb71-9024-4d12-a27a-994892cf38e4-agent-certs\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2dbt\" (UniqueName: \"kubernetes.io/projected/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kube-api-access-r2dbt\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysconfig\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.843713 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-var-lib-kubelet\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82ecc097-1c88-4f8c-928c-708b9a628a72-iptables-alerter-script\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjrr\" (UniqueName: \"kubernetes.io/projected/7a428dff-76d1-44aa-864a-2532953ad0ad-kube-api-access-djjrr\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-tmp\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.843976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.844016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-sys\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.844048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-lib-modules\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.844238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.844090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79d2\" (UniqueName: \"kubernetes.io/projected/82ecc097-1c88-4f8c-928c-708b9a628a72-kube-api-access-z79d2\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.846586 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.846532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.849747 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.849724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.853037 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.853018 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:58.855581 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.855561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:58.867022 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.867000 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:26:58.867120 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.867067 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h5b9m" Oct 31 12:26:58.880157 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.879819 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h5b9m" Oct 31 12:26:58.933430 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.933378 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:26:58.937600 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.937527 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:26:58.937600 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.937568 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:26:58.937759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.937602 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:26:58.938145 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938055 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:26:58.938248 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938178 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:26:58.938248 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938214 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:26:58.938248 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938234 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m5kv7\"" Oct 31 12:26:58.938359 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938266 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:26:58.938359 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938306 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:26:58.938457 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938374 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:26:58.938529 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938490 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:26:58.938594 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938571 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:26:58.938650 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938574 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:26:58.938650 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938607 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:26:58.938650 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938624 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:26:58.938756 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938697 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:26:58.938756 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.938742 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:26:58.939691 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.939671 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9d2j7\"" Oct 31 12:26:58.939773 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.939739 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kldqp\"" Oct 31 12:26:58.939828 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.939771 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-df646\"" Oct 31 12:26:58.940008 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.939991 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzb2l\"" Oct 31 12:26:58.940101 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.940085 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:26:58.944558 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-multus-certs\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.944715 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qglj\" (UniqueName: \"kubernetes.io/projected/ad2b3ee2-f439-49ca-a106-63e231dc232f-kube-api-access-7qglj\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.944715 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.944841 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-tmp\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.944841 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.944841 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:58.945001 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-hostroot\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.945001 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-systemd-units\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945001 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945001 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-kubelet\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945001 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.944987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-systemd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-bin\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:58.945018 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-modprobe-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:58.945084 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs podName:7a428dff-76d1-44aa-864a-2532953ad0ad nodeName:}" failed. No retries permitted until 2026-10-31 12:26:59.445052314 +0000 UTC m=+2.948351346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs") pod "network-metrics-daemon-kr4dj" (UID: "7a428dff-76d1-44aa-864a-2532953ad0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsz9\" (UniqueName: \"kubernetes.io/projected/958a8aee-281c-45d9-8c87-ee2ce835fbee-kube-api-access-qcsz9\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-modprobe-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-device-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945155 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-device-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97b652f-9d17-4078-90ea-ac9bd9820384-ovn-node-metrics-cert\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-registration-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-registration-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-netns\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-kubernetes\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-host\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-kubernetes\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-tuned\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-host\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.945824 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-systemd\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-conf-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945926 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-systemd\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.945991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-host\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32afbb71-9024-4d12-a27a-994892cf38e4-agent-certs\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysconfig\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.946348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjrr\" (UniqueName: \"kubernetes.io/projected/7a428dff-76d1-44aa-864a-2532953ad0ad-kube-api-access-djjrr\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:58.946723 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysconfig\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.946723 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-netns\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.946723 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-log-socket\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.946723 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-serviceca\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:58.946723 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.946723 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-k8s-cni-cncf-io\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6w2l\" (UniqueName: \"kubernetes.io/projected/b626d907-7b4b-4437-9e69-a82333ebf268-kube-api-access-j6w2l\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-netd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-config\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppm8b\" (UniqueName: \"kubernetes.io/projected/d97b652f-9d17-4078-90ea-ac9bd9820384-kube-api-access-ppm8b\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-sys\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-lib-modules\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-sys\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79d2\" (UniqueName: \"kubernetes.io/projected/82ecc097-1c88-4f8c-928c-708b9a628a72-kube-api-access-z79d2\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.946963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-kubelet-config\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947004 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-lib-modules\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.947034 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-etc-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-node-log\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d278316-af3e-4b36-8505-a5d464016e85-tmp-dir\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-run\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-bin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-multus\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-kubelet\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-ovn\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-sys-fs\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-dbus\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-system-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d278316-af3e-4b36-8505-a5d464016e85-hosts-file\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgcz\" (UniqueName: \"kubernetes.io/projected/6d278316-af3e-4b36-8505-a5d464016e85-kube-api-access-zvgcz\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32afbb71-9024-4d12-a27a-994892cf38e4-konnectivity-ca\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-cnibin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-var-lib-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.947716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-script-lib\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-cnibin\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-multus-daemon-config\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-slash\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-socket-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ecc097-1c88-4f8c-928c-708b9a628a72-host-slash\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-socket-dir-parent\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzgv\" (UniqueName: \"kubernetes.io/projected/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-kube-api-access-8vzgv\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.947991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-conf\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dbt\" (UniqueName: \"kubernetes.io/projected/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kube-api-access-r2dbt\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-cni-binary-copy\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-etc-kubernetes\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-env-overrides\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:58.948530 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-system-cni-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-var-lib-kubelet\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82ecc097-1c88-4f8c-928c-708b9a628a72-iptables-alerter-script\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948339 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-var-lib-kubelet\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-run\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-sys-fs\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-socket-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ecc097-1c88-4f8c-928c-708b9a628a72-host-slash\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.948829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-conf\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.949322 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.949058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32afbb71-9024-4d12-a27a-994892cf38e4-konnectivity-ca\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:58.949776 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.949375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-tmp\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.949776 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.949378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82ecc097-1c88-4f8c-928c-708b9a628a72-iptables-alerter-script\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.949776 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.949530 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-tuned\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.949776 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.949629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32afbb71-9024-4d12-a27a-994892cf38e4-agent-certs\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:58.959859 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.959834 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjrr\" (UniqueName: \"kubernetes.io/projected/7a428dff-76d1-44aa-864a-2532953ad0ad-kube-api-access-djjrr\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:58.962308 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.962287 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79d2\" (UniqueName: \"kubernetes.io/projected/82ecc097-1c88-4f8c-928c-708b9a628a72-kube-api-access-z79d2\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:58.962694 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.962677 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsz9\" (UniqueName: \"kubernetes.io/projected/958a8aee-281c-45d9-8c87-ee2ce835fbee-kube-api-access-qcsz9\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:58.965976 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.965946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dbt\" (UniqueName: \"kubernetes.io/projected/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kube-api-access-r2dbt\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:58.970335 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.970283 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"2e9561195d04d47702d33d861addb8e7","Type":"ContainerStarted","Data":"bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091"} Oct 31 12:26:58.972402 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:58.972373 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"d86c20a2519affed9d4fc6daf0172b42","Type":"ContainerStarted","Data":"c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073"} Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-kubelet\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-ovn\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-dbus\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049428 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-kubelet\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-system-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d278316-af3e-4b36-8505-a5d464016e85-hosts-file\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-ovn\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgcz\" (UniqueName: \"kubernetes.io/projected/6d278316-af3e-4b36-8505-a5d464016e85-kube-api-access-zvgcz\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049542 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d278316-af3e-4b36-8505-a5d464016e85-hosts-file\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-cnibin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-var-lib-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-system-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049625 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-dbus\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-var-lib-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-script-lib\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049697 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-cnibin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.049750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-cnibin\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-multus-daemon-config\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049764 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-cnibin\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-slash\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-socket-dir-parent\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzgv\" (UniqueName: \"kubernetes.io/projected/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-kube-api-access-8vzgv\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-cni-binary-copy\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-etc-kubernetes\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.049961 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-env-overrides\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-system-cni-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.050015 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret podName:a166dd5f-2e0c-45a0-8dc6-692741a8e1b3 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:59.549996412 +0000 UTC m=+3.053295423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret") pod "global-pull-secret-syncer-g7zbh" (UID: "a166dd5f-2e0c-45a0-8dc6-692741a8e1b3") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-multus-certs\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-socket-dir-parent\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.050784 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qglj\" (UniqueName: \"kubernetes.io/projected/ad2b3ee2-f439-49ca-a106-63e231dc232f-kube-api-access-7qglj\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.049872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-slash\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-hostroot\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-systemd-units\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-multus-daemon-config\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-kubelet\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-systemd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-bin\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-script-lib\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97b652f-9d17-4078-90ea-ac9bd9820384-ovn-node-metrics-cert\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-netns\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-conf-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.052443 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050524 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-system-cni-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-etc-kubernetes\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050569 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050579 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-multus-certs\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-kubelet\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050612 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-systemd-units\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050631 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-conf-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-hostroot\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-systemd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-host\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-netns\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-netns\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-log-socket\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-bin\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.053444 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-serviceca\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-k8s-cni-cncf-io\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050826 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-log-socket\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6w2l\" (UniqueName: \"kubernetes.io/projected/b626d907-7b4b-4437-9e69-a82333ebf268-kube-api-access-j6w2l\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-netd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-config\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppm8b\" (UniqueName: \"kubernetes.io/projected/d97b652f-9d17-4078-90ea-ac9bd9820384-kube-api-access-ppm8b\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.050985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-kubelet-config\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-etc-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-node-log\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d278316-af3e-4b36-8505-a5d464016e85-tmp-dir\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-bin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-multus\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-multus\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-bin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-env-overrides\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d278316-af3e-4b36-8505-a5d464016e85-tmp-dir\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051470 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-host\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-cni-binary-copy\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-netns\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-netd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051658 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-kubelet-config\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051689 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-etc-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051738 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-k8s-cni-cncf-io\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-node-log\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.051803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-serviceca\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.052589 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-config\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.052819 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.054819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.053776 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97b652f-9d17-4078-90ea-ac9bd9820384-ovn-node-metrics-cert\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.064288 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.064262 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:26:59.064288 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.064289 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:26:59.064418 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.064302 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pw92g for pod openshift-network-diagnostics/network-check-target-zcknp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:26:59.064418 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.064378 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g podName:a632a023-665e-4567-b2fc-13b7c37d152d nodeName:}" failed. No retries permitted until 2026-10-31 12:26:59.564360084 +0000 UTC m=+3.067659112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pw92g" (UniqueName: "kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g") pod "network-check-target-zcknp" (UID: "a632a023-665e-4567-b2fc-13b7c37d152d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:26:59.070578 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.066942 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgcz\" (UniqueName: \"kubernetes.io/projected/6d278316-af3e-4b36-8505-a5d464016e85-kube-api-access-zvgcz\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:59.070578 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.067290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qglj\" (UniqueName: \"kubernetes.io/projected/ad2b3ee2-f439-49ca-a106-63e231dc232f-kube-api-access-7qglj\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.070578 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.067350 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzgv\" (UniqueName: \"kubernetes.io/projected/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-kube-api-access-8vzgv\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:59.070578 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.067665 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6w2l\" (UniqueName: \"kubernetes.io/projected/b626d907-7b4b-4437-9e69-a82333ebf268-kube-api-access-j6w2l\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.070578 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.067715 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppm8b\" (UniqueName: \"kubernetes.io/projected/d97b652f-9d17-4078-90ea-ac9bd9820384-kube-api-access-ppm8b\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.141996 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.141968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:26:59.149638 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.149618 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6b687" Oct 31 12:26:59.152494 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:59.152462 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958a8aee_281c_45d9_8c87_ee2ce835fbee.slice/crio-e74b6a0cc09468c6942fc67ef70989694f1345d231f08d46e3d6f78df9364b79 WatchSource:0}: Error finding container e74b6a0cc09468c6942fc67ef70989694f1345d231f08d46e3d6f78df9364b79: Status 404 returned error can't find the container with id e74b6a0cc09468c6942fc67ef70989694f1345d231f08d46e3d6f78df9364b79 Oct 31 12:26:59.159032 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:59.158996 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32afbb71_9024_4d12_a27a_994892cf38e4.slice/crio-e15db8cdab7ba7373a055383a82face00539d0e5cc00432bad7c3c623fcf5ad8 WatchSource:0}: Error finding container e15db8cdab7ba7373a055383a82face00539d0e5cc00432bad7c3c623fcf5ad8: Status 404 returned error can't find the container with id e15db8cdab7ba7373a055383a82face00539d0e5cc00432bad7c3c623fcf5ad8 Oct 31 12:26:59.163028 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.163008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:26:59.167993 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.167971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:26:59.174869 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:59.174846 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1925581_bb98_4bd2_8c66_5cafa370dfc9.slice/crio-3c0e28ad18a3cf4138d38ee49042564bc6d06463c1742785f511280265b3cc14 WatchSource:0}: Error finding container 3c0e28ad18a3cf4138d38ee49042564bc6d06463c1742785f511280265b3cc14: Status 404 returned error can't find the container with id 3c0e28ad18a3cf4138d38ee49042564bc6d06463c1742785f511280265b3cc14 Oct 31 12:26:59.176383 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.176258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2ppd" Oct 31 12:26:59.178197 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:59.178178 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ecc097_1c88_4f8c_928c_708b9a628a72.slice/crio-aca7a2bbb04671497f30cbb3751e89c0db6e7b379e60eb22f1854b16a8dacac0 WatchSource:0}: Error finding container aca7a2bbb04671497f30cbb3751e89c0db6e7b379e60eb22f1854b16a8dacac0: Status 404 returned error can't find the container with id aca7a2bbb04671497f30cbb3751e89c0db6e7b379e60eb22f1854b16a8dacac0 Oct 31 12:26:59.186224 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:59.186200 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb626d907_7b4b_4437_9e69_a82333ebf268.slice/crio-c68c679ea306addeee1bd266a45a59452fb0ac79e7d0cb157e93aa62a29e0048 WatchSource:0}: Error finding container c68c679ea306addeee1bd266a45a59452fb0ac79e7d0cb157e93aa62a29e0048: Status 404 returned error can't find the container with id c68c679ea306addeee1bd266a45a59452fb0ac79e7d0cb157e93aa62a29e0048 Oct 31 12:26:59.205877 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.205857 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:26:59.207962 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.207941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:26:59.215898 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.215880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:26:59.217010 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:59.216987 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97b652f_9d17_4078_90ea_ac9bd9820384.slice/crio-5c899e55c08191e87dc2980e9d90e93ed51866455f80ef6c79196ad68cfa65f9 WatchSource:0}: Error finding container 5c899e55c08191e87dc2980e9d90e93ed51866455f80ef6c79196ad68cfa65f9: Status 404 returned error can't find the container with id 5c899e55c08191e87dc2980e9d90e93ed51866455f80ef6c79196ad68cfa65f9 Oct 31 12:26:59.222342 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.222322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:26:59.234325 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:26:59.234293 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2b3ee2_f439_49ca_a106_63e231dc232f.slice/crio-1d436ac1dd286e85aa10a55cdbd5f2b547f728710da521dcf6b7476fb0db35d8 WatchSource:0}: Error finding container 1d436ac1dd286e85aa10a55cdbd5f2b547f728710da521dcf6b7476fb0db35d8: Status 404 returned error can't find the container with id 1d436ac1dd286e85aa10a55cdbd5f2b547f728710da521dcf6b7476fb0db35d8 Oct 31 12:26:59.456081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.455989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:59.456232 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.456177 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:26:59.456274 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.456265 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs podName:7a428dff-76d1-44aa-864a-2532953ad0ad nodeName:}" failed. No retries permitted until 2026-10-31 12:27:00.456243817 +0000 UTC m=+3.959542836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs") pod "network-metrics-daemon-kr4dj" (UID: "7a428dff-76d1-44aa-864a-2532953ad0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:26:59.556738 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.556707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:26:59.556921 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.556887 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:26:59.556994 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.556962 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret podName:a166dd5f-2e0c-45a0-8dc6-692741a8e1b3 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:00.556941881 +0000 UTC m=+4.060240892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret") pod "global-pull-secret-syncer-g7zbh" (UID: "a166dd5f-2e0c-45a0-8dc6-692741a8e1b3") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:26:59.657334 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.657300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:26:59.657516 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.657504 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:26:59.657581 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.657526 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:26:59.657581 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.657538 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pw92g for pod openshift-network-diagnostics/network-check-target-zcknp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:26:59.657664 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.657600 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g podName:a632a023-665e-4567-b2fc-13b7c37d152d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:00.657585724 +0000 UTC m=+4.160884730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw92g" (UniqueName: "kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g") pod "network-check-target-zcknp" (UID: "a632a023-665e-4567-b2fc-13b7c37d152d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:26:59.884619 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.884430 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:59 +0000 UTC" deadline="2028-06-28 21:54:57.668517778 +0000 UTC" Oct 31 12:26:59.884619 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.884518 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14553h27m57.784004282s" Oct 31 12:26:59.964708 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.964675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:26:59.964920 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:26:59.964824 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:26:59.994760 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:26:59.994726 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjlsm" event={"ID":"6d278316-af3e-4b36-8505-a5d464016e85","Type":"ContainerStarted","Data":"c742a42beed1730c1859e9d6b22dadf86deb83a71a721a68b5cba1916362961f"} Oct 31 12:27:00.004175 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.004143 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"5c899e55c08191e87dc2980e9d90e93ed51866455f80ef6c79196ad68cfa65f9"} Oct 31 12:27:00.018839 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.018810 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2ppd" event={"ID":"b626d907-7b4b-4437-9e69-a82333ebf268","Type":"ContainerStarted","Data":"c68c679ea306addeee1bd266a45a59452fb0ac79e7d0cb157e93aa62a29e0048"} Oct 31 12:27:00.023173 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.023099 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6b687" event={"ID":"32afbb71-9024-4d12-a27a-994892cf38e4","Type":"ContainerStarted","Data":"e15db8cdab7ba7373a055383a82face00539d0e5cc00432bad7c3c623fcf5ad8"} Oct 31 12:27:00.030435 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.030377 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" event={"ID":"958a8aee-281c-45d9-8c87-ee2ce835fbee","Type":"ContainerStarted","Data":"e74b6a0cc09468c6942fc67ef70989694f1345d231f08d46e3d6f78df9364b79"} Oct 31 12:27:00.032667 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.032561 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerStarted","Data":"1d436ac1dd286e85aa10a55cdbd5f2b547f728710da521dcf6b7476fb0db35d8"} Oct 31 12:27:00.044782 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.044758 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dz7qv" event={"ID":"8fc0c5ec-9758-40ff-9990-ab568b8d02dd","Type":"ContainerStarted","Data":"b3950d83734febb3fbfa39c130771e6be499b0cfe63820b169f74db1d5e37c98"} Oct 31 12:27:00.048707 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.048683 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2wpv" event={"ID":"82ecc097-1c88-4f8c-928c-708b9a628a72","Type":"ContainerStarted","Data":"aca7a2bbb04671497f30cbb3751e89c0db6e7b379e60eb22f1854b16a8dacac0"} Oct 31 12:27:00.051451 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.051425 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" event={"ID":"b1925581-bb98-4bd2-8c66-5cafa370dfc9","Type":"ContainerStarted","Data":"3c0e28ad18a3cf4138d38ee49042564bc6d06463c1742785f511280265b3cc14"} Oct 31 12:27:00.466323 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.466149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:00.466484 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.466322 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:00.466484 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.466397 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs podName:7a428dff-76d1-44aa-864a-2532953ad0ad nodeName:}" failed. No retries permitted until 2026-10-31 12:27:02.466375432 +0000 UTC m=+5.969674443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs") pod "network-metrics-daemon-kr4dj" (UID: "7a428dff-76d1-44aa-864a-2532953ad0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:00.567728 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.567035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:00.567728 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.567196 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:00.567728 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.567300 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret podName:a166dd5f-2e0c-45a0-8dc6-692741a8e1b3 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:02.567240812 +0000 UTC m=+6.070539833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret") pod "global-pull-secret-syncer-g7zbh" (UID: "a166dd5f-2e0c-45a0-8dc6-692741a8e1b3") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:00.668104 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.668069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:00.668280 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.668244 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:27:00.668280 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.668265 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:27:00.668280 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.668278 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pw92g for pod openshift-network-diagnostics/network-check-target-zcknp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:00.668439 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.668335 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g podName:a632a023-665e-4567-b2fc-13b7c37d152d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:02.668317247 +0000 UTC m=+6.171616271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw92g" (UniqueName: "kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g") pod "network-check-target-zcknp" (UID: "a632a023-665e-4567-b2fc-13b7c37d152d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:00.885418 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.885261 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:59 +0000 UTC" deadline="2028-07-05 12:18:45.265927666 +0000 UTC" Oct 31 12:27:00.885418 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.885301 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14711h51m44.380632594s" Oct 31 12:27:00.967002 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.966125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:00.967002 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.966473 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:00.967002 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:00.966574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:00.967002 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:00.966656 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:01.070961 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:01.070928 2578 generic.go:356] "Generic (PLEG): container finished" podID="d86c20a2519affed9d4fc6daf0172b42" containerID="7e758f589390f8bcae257e4c24d0d7f8679b06ecdcf5eedfd7a9d06bb9ad441c" exitCode=0 Oct 31 12:27:01.071144 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:01.070986 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"d86c20a2519affed9d4fc6daf0172b42","Type":"ContainerDied","Data":"7e758f589390f8bcae257e4c24d0d7f8679b06ecdcf5eedfd7a9d06bb9ad441c"} Oct 31 12:27:01.964137 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:01.964026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:01.964699 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:01.964179 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:02.094425 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:02.093554 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"d86c20a2519affed9d4fc6daf0172b42","Type":"ContainerStarted","Data":"2f9edf796eb9ca6a7d35f0a4b3fda0d378fdd1875cf11d84c9b5787daceacabd"} Oct 31 12:27:02.114483 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:02.114420 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" podStartSLOduration=3.11439869 podStartE2EDuration="3.11439869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:02.113584416 +0000 UTC m=+5.616883450" watchObservedRunningTime="2026-10-31 12:27:02.11439869 +0000 UTC m=+5.617697723" Oct 31 12:27:02.484607 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:02.484523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:02.484837 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.484795 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:02.484918 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.484867 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs podName:7a428dff-76d1-44aa-864a-2532953ad0ad nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.484846649 +0000 UTC m=+9.988145677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs") pod "network-metrics-daemon-kr4dj" (UID: "7a428dff-76d1-44aa-864a-2532953ad0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:02.586218 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:02.585595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:02.586218 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.585751 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:02.586218 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.585806 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret podName:a166dd5f-2e0c-45a0-8dc6-692741a8e1b3 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.585789784 +0000 UTC m=+10.089088809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret") pod "global-pull-secret-syncer-g7zbh" (UID: "a166dd5f-2e0c-45a0-8dc6-692741a8e1b3") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:02.686846 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:02.686256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:02.686846 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.686369 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:27:02.686846 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.686382 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:27:02.686846 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.686391 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pw92g for pod openshift-network-diagnostics/network-check-target-zcknp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:02.686846 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.686429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g podName:a632a023-665e-4567-b2fc-13b7c37d152d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.686415953 +0000 UTC m=+10.189714960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw92g" (UniqueName: "kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g") pod "network-check-target-zcknp" (UID: "a632a023-665e-4567-b2fc-13b7c37d152d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:02.967189 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:02.966500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:02.967189 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.966647 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:02.967189 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:02.966731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:02.967189 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:02.966807 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:03.104598 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:03.104510 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"2e9561195d04d47702d33d861addb8e7","Type":"ContainerStarted","Data":"61996ca91cb34a11b03f37f9d73e12798a624d90de32e9b5ebe4131ac3a3b602"} Oct 31 12:27:03.109776 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:03.109744 2578 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="a8ad48553cb66f49747c49a7d5a85c4f3a8ad146659f322ed7f8ce72cfa52284" exitCode=0 Oct 31 12:27:03.110832 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:03.110808 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerDied","Data":"a8ad48553cb66f49747c49a7d5a85c4f3a8ad146659f322ed7f8ce72cfa52284"} Oct 31 12:27:03.153595 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:03.152541 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" podStartSLOduration=4.152523129 podStartE2EDuration="4.152523129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:03.127232085 +0000 UTC m=+6.630531115" watchObservedRunningTime="2026-10-31 12:27:03.152523129 +0000 UTC m=+6.655822155" Oct 31 12:27:03.963495 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:03.963450 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:03.963643 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:03.963613 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:04.124643 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:04.124612 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6b687" event={"ID":"32afbb71-9024-4d12-a27a-994892cf38e4","Type":"ContainerStarted","Data":"8b30492c6b1a97c57566ccfa94569eaa927d2ce5f38fdc30f245b4136a7bba7c"} Oct 31 12:27:04.968432 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:04.967724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:04.968432 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:04.967910 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:04.969921 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:04.968773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:04.969921 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:04.968925 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:05.129786 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:05.129749 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dz7qv" event={"ID":"8fc0c5ec-9758-40ff-9990-ab568b8d02dd","Type":"ContainerStarted","Data":"89a51ba7cc98855aaa305f4e1efb31e7bda24e0e8a28a2cc61c81b9e0b9fa8ce"} Oct 31 12:27:05.156318 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:05.156262 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dz7qv" podStartSLOduration=2.810688547 podStartE2EDuration="8.156245449s" totalImagesPullingTime="5.345556902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:05.155516034 +0000 UTC m=+8.658815068" watchObservedRunningTime="2026-10-31 12:27:05.156245449 +0000 UTC m=+8.659544481" Oct 31 12:27:05.156512 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:05.156375 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6b687" podStartSLOduration=3.433088082 podStartE2EDuration="8.156364711s" totalImagesPullingTime="4.723276629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:04.14462991 +0000 UTC m=+7.647928945" watchObservedRunningTime="2026-10-31 12:27:05.156364711 +0000 UTC m=+8.659663742" Oct 31 12:27:05.159925 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:05.159832 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:05.359721 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:05.359646 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:05.964037 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:05.963959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:05.964211 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:05.964117 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:06.134887 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:06.134851 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" event={"ID":"b1925581-bb98-4bd2-8c66-5cafa370dfc9","Type":"ContainerStarted","Data":"2355ac3d185315dcdcf8bb579081dc6746d2dd9401fc4093d07fe6fff8c9c022"} Oct 31 12:27:06.519323 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:06.519286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:06.519518 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.519485 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:06.519604 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.519571 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs podName:7a428dff-76d1-44aa-864a-2532953ad0ad nodeName:}" failed. No retries permitted until 2026-10-31 12:27:14.51955182 +0000 UTC m=+18.022850829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs") pod "network-metrics-daemon-kr4dj" (UID: "7a428dff-76d1-44aa-864a-2532953ad0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:06.621331 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:06.620675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:06.621331 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.620872 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:06.621331 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.620938 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret podName:a166dd5f-2e0c-45a0-8dc6-692741a8e1b3 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:14.620919082 +0000 UTC m=+18.124218089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret") pod "global-pull-secret-syncer-g7zbh" (UID: "a166dd5f-2e0c-45a0-8dc6-692741a8e1b3") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:06.721529 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:06.721397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:06.721701 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.721630 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:27:06.721701 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.721665 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:27:06.721701 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.721679 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pw92g for pod openshift-network-diagnostics/network-check-target-zcknp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:06.721879 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.721734 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g podName:a632a023-665e-4567-b2fc-13b7c37d152d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:14.721716612 +0000 UTC m=+18.225015622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw92g" (UniqueName: "kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g") pod "network-check-target-zcknp" (UID: "a632a023-665e-4567-b2fc-13b7c37d152d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:06.967422 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:06.965879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:06.967422 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.965982 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:06.967422 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:06.966026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:06.967422 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:06.966068 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:07.323024 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.322991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:07.453459 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.453432 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6b687" Oct 31 12:27:07.454791 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.454580 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6b687" Oct 31 12:27:07.457348 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.457322 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:07.468756 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.468260 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:27:07.905024 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.904902 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:27:07.468277513Z","UUID":"e688487f-5c2d-4dfb-8287-7dc289dc87ef","Handler":null,"Name":"","Endpoint":""} Oct 31 12:27:07.906763 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.906739 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:27:07.906883 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.906770 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:27:07.964136 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:07.964113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:07.964270 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:07.964241 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:08.142924 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.142844 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" event={"ID":"b1925581-bb98-4bd2-8c66-5cafa370dfc9","Type":"ContainerStarted","Data":"19862d233ec1dd11cd965d35da2a6cd57f9004fb6a9def23b268f7e0e25b7a96"} Oct 31 12:27:08.146150 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.145697 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjlsm" event={"ID":"6d278316-af3e-4b36-8505-a5d464016e85","Type":"ContainerStarted","Data":"936cce58029cef8b06db817e29147418c6a99e3c1a85ee58e0ee2c4af3df47ac"} Oct 31 12:27:08.148433 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.148396 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2ppd" event={"ID":"b626d907-7b4b-4437-9e69-a82333ebf268","Type":"ContainerStarted","Data":"1fe8c34e9139a8cc9abc7e366ae0e89853e5a506af36b3f94e1bd65c745c8022"} Oct 31 12:27:08.148574 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.148446 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6b687" Oct 31 12:27:08.148731 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.148712 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6b687" Oct 31 12:27:08.164688 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.164640 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vjlsm" podStartSLOduration=2.8711870790000003 podStartE2EDuration="11.164623675s" totalImagesPullingTime="8.293436596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:08.164290735 +0000 UTC m=+11.667589782" watchObservedRunningTime="2026-10-31 12:27:08.164623675 +0000 UTC m=+11.667922712" Oct 31 12:27:08.185850 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.185802 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-p2ppd" podStartSLOduration=3.18038117 podStartE2EDuration="11.185787557s" totalImagesPullingTime="8.005406387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:08.185611987 +0000 UTC m=+11.688911018" watchObservedRunningTime="2026-10-31 12:27:08.185787557 +0000 UTC m=+11.689086588" Oct 31 12:27:08.313459 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.313267 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:08.445722 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.445649 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:08.963848 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.963814 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:08.963848 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:08.963859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:08.964076 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:08.963986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:08.964322 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:08.964266 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:09.106634 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:09.106575 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:09.152500 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:09.152450 2578 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="a50140f0da59749fe2a68810e7757f6feaf607437e2228bddf75d1a793e1265c" exitCode=0 Oct 31 12:27:09.152500 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:09.152494 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerDied","Data":"a50140f0da59749fe2a68810e7757f6feaf607437e2228bddf75d1a793e1265c"} Oct 31 12:27:09.156111 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:09.156054 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" event={"ID":"b1925581-bb98-4bd2-8c66-5cafa370dfc9","Type":"ContainerStarted","Data":"9e025921a4b9326ace9fcadc9508807a4ee87abf3ed6044973cd8ed74a3ca0c0"} Oct 31 12:27:09.202990 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:09.202953 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" podStartSLOduration=3.028980651 podStartE2EDuration="12.202937457s" totalImagesPullingTime="9.173956806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:09.202394189 +0000 UTC m=+12.705693228" watchObservedRunningTime="2026-10-31 12:27:09.202937457 +0000 UTC m=+12.706236487" Oct 31 12:27:09.238138 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:09.238112 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:09.963940 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:09.963911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:09.964528 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:09.964030 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:10.160447 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.160423 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerStarted","Data":"dd9bda1f4a04a917f1eb060a7b30f5a112d52ccfc5b8757e8de71960f1d169ea"} Oct 31 12:27:10.161690 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.161664 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2wpv" event={"ID":"82ecc097-1c88-4f8c-928c-708b9a628a72","Type":"ContainerStarted","Data":"ff2bd2161029fdfccd62a9bdd773cf4d5089111b3b4e4f47d5048f8787dd4bba"} Oct 31 12:27:10.163045 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.163018 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" event={"ID":"958a8aee-281c-45d9-8c87-ee2ce835fbee","Type":"ContainerStarted","Data":"28175f013c1aea1a4dc7662cd090cee773a43de3498f245dd4073eb9f728dcb5"} Oct 31 12:27:10.178028 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.177991 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s2wpv" podStartSLOduration=4.511249141 podStartE2EDuration="13.177979784s" totalImagesPullingTime="8.666730643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:10.177773188 +0000 UTC m=+13.681072215" watchObservedRunningTime="2026-10-31 12:27:10.177979784 +0000 UTC m=+13.681278810" Oct 31 12:27:10.197226 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.197179 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" podStartSLOduration=2.979620541 podStartE2EDuration="13.197166008s" totalImagesPullingTime="10.217545467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:10.196723141 +0000 UTC m=+13.700022164" watchObservedRunningTime="2026-10-31 12:27:10.197166008 +0000 UTC m=+13.700465036" Oct 31 12:27:10.334032 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.333975 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:10.470116 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.470089 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:10.964268 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.964239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:10.964858 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:10.964391 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:10.964858 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:10.964427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:10.964858 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:10.964522 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:11.027360 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:11.027334 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:11.153268 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:11.153246 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:11.167009 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:11.166982 2578 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="dd9bda1f4a04a917f1eb060a7b30f5a112d52ccfc5b8757e8de71960f1d169ea" exitCode=0 Oct 31 12:27:11.167945 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:11.167922 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerDied","Data":"dd9bda1f4a04a917f1eb060a7b30f5a112d52ccfc5b8757e8de71960f1d169ea"} Oct 31 12:27:11.308003 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:11.307927 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:11.434409 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:11.434370 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:11.963410 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:11.963382 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:11.963559 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:11.963491 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:12.041663 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:12.041635 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:12.163504 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:12.163312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:12.749059 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:12.749019 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:12.884141 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:12.884117 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:12.964368 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:12.964348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:12.964478 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:12.964443 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:12.964531 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:12.964500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:12.964567 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:12.964552 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:13.016815 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:13.016767 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:13.172057 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:13.172033 2578 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="3c58748ca8f48a956507ee3820dcb63edb2d8bdd6cac16af9f68507f507dbd36" exitCode=0 Oct 31 12:27:13.172419 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:13.172066 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerDied","Data":"3c58748ca8f48a956507ee3820dcb63edb2d8bdd6cac16af9f68507f507dbd36"} Oct 31 12:27:13.964234 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:13.964199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:13.964413 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:13.964336 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:14.020351 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.020308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:14.571256 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.571224 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:14.579310 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.579290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:14.579421 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.579393 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:14.579474 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.579437 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs podName:7a428dff-76d1-44aa-864a-2532953ad0ad nodeName:}" failed. No retries permitted until 2026-10-31 12:27:30.579425968 +0000 UTC m=+34.082724973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs") pod "network-metrics-daemon-kr4dj" (UID: "7a428dff-76d1-44aa-864a-2532953ad0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:14.679690 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.679662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:14.679795 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.679752 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:14.679795 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.679793 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret podName:a166dd5f-2e0c-45a0-8dc6-692741a8e1b3 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:30.679783163 +0000 UTC m=+34.183082169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret") pod "global-pull-secret-syncer-g7zbh" (UID: "a166dd5f-2e0c-45a0-8dc6-692741a8e1b3") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:14.727584 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.727553 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:14.780653 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.780621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:14.780761 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.780746 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:27:14.780761 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.780761 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:27:14.780848 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.780770 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pw92g for pod openshift-network-diagnostics/network-check-target-zcknp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:14.780848 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.780814 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g podName:a632a023-665e-4567-b2fc-13b7c37d152d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:30.780802083 +0000 UTC m=+34.284101101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw92g" (UniqueName: "kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g") pod "network-check-target-zcknp" (UID: "a632a023-665e-4567-b2fc-13b7c37d152d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:14.846159 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.846089 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjlsm_6d278316-af3e-4b36-8505-a5d464016e85/dns-node-resolver/0.log" Oct 31 12:27:14.853320 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.853296 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:14.964235 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.963860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:14.964235 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.963968 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:14.964425 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.964341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:14.964509 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:14.964451 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:14.995637 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:14.995611 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:15.134602 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:15.134535 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:15.446404 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:15.446330 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dz7qv_8fc0c5ec-9758-40ff-9990-ab568b8d02dd/node-ca/0.log" Oct 31 12:27:15.963992 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:15.963963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:15.964442 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:15.964094 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:16.007170 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.007116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:16.134403 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.134361 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:16.180741 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.180716 2578 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="1d48bc66fae186e82afa50e31a748128531045c389d151baf59431facc71c819" exitCode=0 Oct 31 12:27:16.180869 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.180767 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerDied","Data":"1d48bc66fae186e82afa50e31a748128531045c389d151baf59431facc71c819"} Oct 31 12:27:16.517068 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.517035 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:16.706359 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.706310 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:16.916452 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.916424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:16.964621 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.964544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:16.965004 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:16.964641 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:16.965065 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:16.965046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:16.965157 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:16.965136 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:17.185931 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:17.185897 2578 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="83cb5584f2eb69184127e902ce3c77c7e77dba7a973cfc694989f669b8245ca7" exitCode=0 Oct 31 12:27:17.186067 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:17.185943 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerDied","Data":"83cb5584f2eb69184127e902ce3c77c7e77dba7a973cfc694989f669b8245ca7"} Oct 31 12:27:17.310925 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:17.310898 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:17.842060 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:17.842016 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:17.963531 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:17.963503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:17.963698 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:17.963620 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:18.115970 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.115892 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:18.192323 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.192298 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" event={"ID":"ad2b3ee2-f439-49ca-a106-63e231dc232f","Type":"ContainerStarted","Data":"c591f26fc63439d4c8f6fe54a2af96f6ecaac18bbe97fc16cbfd350ab8c30bce"} Oct 31 12:27:18.246012 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.245966 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wnnvc" podStartSLOduration=8.504067502 podStartE2EDuration="21.245948621s" totalImagesPullingTime="12.741881119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:18.244091707 +0000 UTC m=+21.747390733" watchObservedRunningTime="2026-10-31 12:27:18.245948621 +0000 UTC m=+21.749247648" Oct 31 12:27:18.507718 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.507650 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:18.710548 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.710527 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:18.908558 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.908533 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:18.963907 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.963878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:18.964012 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:18.963997 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:18.964081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:18.964045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:18.964148 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:18.964129 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:19.197995 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.197965 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"7dfe2eb2d173a41ca6bc37ad5baf2cda6b191bc0d5b094c8640559c6e030cfd8"} Oct 31 12:27:19.198422 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.198030 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"8859b0f0588e0189b0254ccc73105e17fda354113496546da1607cf5670c2179"} Oct 31 12:27:19.198422 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.198075 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"789c18804a522406164b6b125e03c35fdadef2b0a38d797bd15a1cdadba7d02e"} Oct 31 12:27:19.198422 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.198105 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"4f82142146d9d56b36e912b415e8cc4c99bb6ec98c3608a4f0cf4860e44f321e"} Oct 31 12:27:19.198422 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.198135 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"f45889bcd45bfb4f6ba26835b2866afe8b6f984b4562df6bfe671489cc454dd2"} Oct 31 12:27:19.198422 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.198171 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"e87942df3c52fffba8ea6e596f36203b34e68e67149f000cb037216e0f100612"} Oct 31 12:27:19.309861 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.309835 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:19.567905 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.567843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:19.915229 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.915205 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:19.963864 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:19.963847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:19.963952 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:19.963935 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:20.508087 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:20.508063 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:20.730786 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:20.730758 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:20.964375 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:20.964353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:20.964551 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:20.964451 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:20.964551 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:20.964514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:20.964665 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:20.964622 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:21.125594 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:21.125568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:21.204208 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:21.204154 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"98f1114da6746806d10b64beafbcd2daa73e6294beb59b2a1c96d31d7e35b1e9"} Oct 31 12:27:21.305418 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:21.305392 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:21.909140 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:21.909118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:21.964257 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:21.964239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:21.964353 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:21.964335 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:22.310718 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:22.310643 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:22.445764 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:22.445736 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:22.717141 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:22.717108 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:22.966346 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:22.966311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:22.966745 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:22.966525 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:22.967008 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:22.966982 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:22.967140 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:22.967116 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:23.115682 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.115653 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:23.212239 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.212199 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" event={"ID":"d97b652f-9d17-4078-90ea-ac9bd9820384","Type":"ContainerStarted","Data":"d70736d9dea8706073277ee7857081f64b6861a02127e278ec99bfdc02949a1d"} Oct 31 12:27:23.212503 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.212475 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:27:23.212654 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.212565 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:27:23.227670 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.227650 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:27:23.241158 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.241108 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:23.247381 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.247342 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" podStartSLOduration=6.9070293209999996 podStartE2EDuration="26.247331766s" totalImagesPullingTime="19.340302445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:23.245276894 +0000 UTC m=+26.748575921" watchObservedRunningTime="2026-10-31 12:27:23.247331766 +0000 UTC m=+26.750630792" Oct 31 12:27:23.510574 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.510497 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:23.910823 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.910793 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:23.964081 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:23.964058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:23.964182 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:23.964159 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:24.214542 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:24.214475 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:27:24.230538 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:24.230513 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:27:24.314338 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:24.314311 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:24.714393 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:24.714367 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:24.913581 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:24.913558 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:24.963983 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:24.963962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:24.964094 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:24.964073 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:24.964159 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:24.964134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:24.964222 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:24.964206 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:25.109323 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:25.109299 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:25.236840 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:25.236815 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:25.659274 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:25.659246 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:25.963540 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:25.963517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:25.963637 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:25.963616 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:25.975621 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:25.975592 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:26.103950 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:26.103923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:26.964898 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:26.964872 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:26.965242 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:26.964946 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:26.965242 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:26.965040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:26.965242 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:26.965123 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:27.964244 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:27.964216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:27.964407 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:27.964311 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:28.963975 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:28.963939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:28.964438 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:28.964058 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:28.964438 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:28.964110 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:28.964438 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:28.964201 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:29.667913 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:29.667857 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:29.796849 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:29.796810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:29.927208 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:29.927143 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:29.963941 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:29.963919 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:29.964039 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:29.964022 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:30.053950 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:30.053924 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:30.367418 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:30.367397 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:30.590498 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:30.590458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:30.590638 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.590617 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:30.590689 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.590683 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs podName:7a428dff-76d1-44aa-864a-2532953ad0ad nodeName:}" failed. No retries permitted until 2026-10-31 12:28:02.590667334 +0000 UTC m=+66.093966387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs") pod "network-metrics-daemon-kr4dj" (UID: "7a428dff-76d1-44aa-864a-2532953ad0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:27:30.691063 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:30.691013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:30.691167 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.691149 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:30.691216 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.691205 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret podName:a166dd5f-2e0c-45a0-8dc6-692741a8e1b3 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:02.691191609 +0000 UTC m=+66.194490615 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret") pod "global-pull-secret-syncer-g7zbh" (UID: "a166dd5f-2e0c-45a0-8dc6-692741a8e1b3") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:27:30.792055 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:30.792034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:30.792150 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.792130 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:27:30.792150 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.792145 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:27:30.792225 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.792154 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pw92g for pod openshift-network-diagnostics/network-check-target-zcknp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:30.792225 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.792193 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g podName:a632a023-665e-4567-b2fc-13b7c37d152d nodeName:}" failed. No retries permitted until 2026-10-31 12:28:02.79218263 +0000 UTC m=+66.295481649 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw92g" (UniqueName: "kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g") pod "network-check-target-zcknp" (UID: "a632a023-665e-4567-b2fc-13b7c37d152d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:27:30.963630 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:30.963570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:30.963630 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:30.963599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:30.963762 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.963662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:30.963827 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:30.963805 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:31.053193 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.053141 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:31.223725 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.223622 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:31.392379 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.392340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:31.527170 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.527098 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:31.791524 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.791431 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7zbh"] Oct 31 12:27:31.791700 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.791610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:31.791770 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:31.791718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:31.795116 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.795091 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcknp"] Oct 31 12:27:31.795315 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.795296 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kr4dj"] Oct 31 12:27:31.795405 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.795389 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:31.795521 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:31.795501 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:31.795633 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.795585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:31.795761 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:31.795694 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:31.944756 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:31.944727 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:32.074196 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:32.074175 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:32.306326 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:32.306301 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:32.438677 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:32.438617 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:33.456396 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:33.456337 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjlsm_6d278316-af3e-4b36-8505-a5d464016e85/dns-node-resolver/0.log" Oct 31 12:27:33.963991 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:33.963960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:33.964172 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:33.964104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:33.964172 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:33.964127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:33.964299 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:33.964175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:33.964299 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:33.964207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:33.964398 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:33.964304 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:33.983582 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:33.983558 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:34.112317 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:34.112294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:34.256414 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:34.256358 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dz7qv_8fc0c5ec-9758-40ff-9990-ab568b8d02dd/node-ca/0.log" Oct 31 12:27:35.963669 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:35.963634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:35.963669 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:35.963662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:35.964052 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:35.963747 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:35.964052 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:35.963748 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcknp" podUID="a632a023-665e-4567-b2fc-13b7c37d152d" Oct 31 12:27:35.964052 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:35.963837 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kr4dj" podUID="7a428dff-76d1-44aa-864a-2532953ad0ad" Oct 31 12:27:35.964052 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:27:35.963919 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7zbh" podUID="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" Oct 31 12:27:36.849024 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.848997 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeReady" Oct 31 12:27:36.849141 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.849102 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 12:27:36.897691 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.897622 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4jx6k"] Oct 31 12:27:36.912376 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.912349 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b6x24"] Oct 31 12:27:36.912493 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.912433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:36.926543 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.926379 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4jx6k"] Oct 31 12:27:36.926648 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.926561 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b6x24"] Oct 31 12:27:36.926648 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.926512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:36.927653 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.927631 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jlbzw"] Oct 31 12:27:36.938490 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.938446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:36.941356 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.941336 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jlbzw"] Oct 31 12:27:36.948555 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.948536 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:27:36.948965 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.948920 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c6kzr\"" Oct 31 12:27:36.949078 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.949007 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952030 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcqch\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952052 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nzzrg\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952064 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952073 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952112 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952126 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952127 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952137 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:27:36.952152 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.952114 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:27:36.981153 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:36.981130 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:37.047175 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2726bde3-96ae-4717-8732-f7e792a0ddfb-crio-socket\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.047175 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2726bde3-96ae-4717-8732-f7e792a0ddfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.047320 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/052e1a61-3220-4a91-83c5-a08bb829f358-metrics-tls\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.047320 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdpk5\" (UniqueName: \"kubernetes.io/projected/052e1a61-3220-4a91-83c5-a08bb829f358-kube-api-access-qdpk5\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.047401 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2726bde3-96ae-4717-8732-f7e792a0ddfb-data-volume\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.047401 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.047401 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9c6\" (UniqueName: \"kubernetes.io/projected/c41fab2e-6e79-4fe1-a46e-996b4012efdf-kube-api-access-wk9c6\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:37.047533 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c41fab2e-6e79-4fe1-a46e-996b4012efdf-cert\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:37.047533 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7tf9\" (UniqueName: \"kubernetes.io/projected/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-api-access-p7tf9\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.047533 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/052e1a61-3220-4a91-83c5-a08bb829f358-config-volume\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.047634 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.047542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052e1a61-3220-4a91-83c5-a08bb829f358-tmp-dir\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.122789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.122767 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:37.147903 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.147851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdpk5\" (UniqueName: \"kubernetes.io/projected/052e1a61-3220-4a91-83c5-a08bb829f358-kube-api-access-qdpk5\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.147994 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.147902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2726bde3-96ae-4717-8732-f7e792a0ddfb-data-volume\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.147994 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.147926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.147994 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.147956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9c6\" (UniqueName: \"kubernetes.io/projected/c41fab2e-6e79-4fe1-a46e-996b4012efdf-kube-api-access-wk9c6\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:37.148131 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.147992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c41fab2e-6e79-4fe1-a46e-996b4012efdf-cert\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:37.148131 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7tf9\" (UniqueName: \"kubernetes.io/projected/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-api-access-p7tf9\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.148131 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/052e1a61-3220-4a91-83c5-a08bb829f358-config-volume\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.148131 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052e1a61-3220-4a91-83c5-a08bb829f358-tmp-dir\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.148131 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2726bde3-96ae-4717-8732-f7e792a0ddfb-crio-socket\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.148360 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2726bde3-96ae-4717-8732-f7e792a0ddfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.148360 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/052e1a61-3220-4a91-83c5-a08bb829f358-metrics-tls\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.148651 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148620 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2726bde3-96ae-4717-8732-f7e792a0ddfb-crio-socket\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.149050 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.148979 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052e1a61-3220-4a91-83c5-a08bb829f358-tmp-dir\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.149220 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.149199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.149512 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.149318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2726bde3-96ae-4717-8732-f7e792a0ddfb-data-volume\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.149874 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.149790 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/052e1a61-3220-4a91-83c5-a08bb829f358-config-volume\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.156570 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.153143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/052e1a61-3220-4a91-83c5-a08bb829f358-metrics-tls\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.156570 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.153146 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2726bde3-96ae-4717-8732-f7e792a0ddfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.156570 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.153253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c41fab2e-6e79-4fe1-a46e-996b4012efdf-cert\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:37.160500 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.160453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9c6\" (UniqueName: \"kubernetes.io/projected/c41fab2e-6e79-4fe1-a46e-996b4012efdf-kube-api-access-wk9c6\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:37.160760 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.160744 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdpk5\" (UniqueName: \"kubernetes.io/projected/052e1a61-3220-4a91-83c5-a08bb829f358-kube-api-access-qdpk5\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.161647 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.161628 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7tf9\" (UniqueName: \"kubernetes.io/projected/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-api-access-p7tf9\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.253145 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.253128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:37.269593 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.269574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:27:37.310754 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.310601 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:27:37.313786 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.313760 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:37.442303 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.442248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:37.581935 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.581898 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:37.604773 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.604420 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jlbzw"] Oct 31 12:27:37.607353 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:37.607329 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41fab2e_6e79_4fe1_a46e_996b4012efdf.slice/crio-f9a67f7b03a671338555684404a3efa934cefa0e6e14f71c12515c7b624bfded WatchSource:0}: Error finding container f9a67f7b03a671338555684404a3efa934cefa0e6e14f71c12515c7b624bfded: Status 404 returned error can't find the container with id f9a67f7b03a671338555684404a3efa934cefa0e6e14f71c12515c7b624bfded Oct 31 12:27:37.607678 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.607644 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4jx6k"] Oct 31 12:27:37.610109 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:37.610084 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052e1a61_3220_4a91_83c5_a08bb829f358.slice/crio-d3b15170a311240f888e8ec6647dec45c00df493eababc9b7b485e847c28cf22 WatchSource:0}: Error finding container d3b15170a311240f888e8ec6647dec45c00df493eababc9b7b485e847c28cf22: Status 404 returned error can't find the container with id d3b15170a311240f888e8ec6647dec45c00df493eababc9b7b485e847c28cf22 Oct 31 12:27:37.610854 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.610836 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b6x24"] Oct 31 12:27:37.613615 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:37.613593 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2726bde3_96ae_4717_8732_f7e792a0ddfb.slice/crio-82c724bb90c5850cdda5082faa02beba25df46a91866906e87b98d72f7f0fb83 WatchSource:0}: Error finding container 82c724bb90c5850cdda5082faa02beba25df46a91866906e87b98d72f7f0fb83: Status 404 returned error can't find the container with id 82c724bb90c5850cdda5082faa02beba25df46a91866906e87b98d72f7f0fb83 Oct 31 12:27:37.712608 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.712584 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:37.963661 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.963577 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:27:37.963781 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.963708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:27:37.963883 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.963865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:27:37.970704 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.970680 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:27:37.972287 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.972265 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\"" Oct 31 12:27:37.972405 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.972272 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\"" Oct 31 12:27:37.972405 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.972299 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\"" Oct 31 12:27:37.972405 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.972307 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:27:37.972405 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.972285 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:27:37.972405 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:37.972299 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:27:38.244535 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:38.244484 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jlbzw" event={"ID":"c41fab2e-6e79-4fe1-a46e-996b4012efdf","Type":"ContainerStarted","Data":"f9a67f7b03a671338555684404a3efa934cefa0e6e14f71c12515c7b624bfded"} Oct 31 12:27:38.245636 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:38.245618 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4jx6k" event={"ID":"052e1a61-3220-4a91-83c5-a08bb829f358","Type":"ContainerStarted","Data":"d3b15170a311240f888e8ec6647dec45c00df493eababc9b7b485e847c28cf22"} Oct 31 12:27:38.246960 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:38.246942 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6x24" event={"ID":"2726bde3-96ae-4717-8732-f7e792a0ddfb","Type":"ContainerStarted","Data":"d4a37cf76b15386ad5478b2e382a2e5876390ebd101a41204a783626529bb077"} Oct 31 12:27:38.247030 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:38.246978 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6x24" event={"ID":"2726bde3-96ae-4717-8732-f7e792a0ddfb","Type":"ContainerStarted","Data":"82c724bb90c5850cdda5082faa02beba25df46a91866906e87b98d72f7f0fb83"} Oct 31 12:27:39.199814 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.199793 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:39.251144 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.251116 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6x24" event={"ID":"2726bde3-96ae-4717-8732-f7e792a0ddfb","Type":"ContainerStarted","Data":"be6c2ace00c40105b3ed0c554d2a65cd023fd444c8419c0157baf02c3ea1e6ee"} Oct 31 12:27:39.335108 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.335076 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl"] Oct 31 12:27:39.361434 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.361386 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl"] Oct 31 12:27:39.361542 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.361501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.363768 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.363528 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw"] Oct 31 12:27:39.369413 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.369151 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:27:39.369413 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.369253 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:27:39.369543 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.369444 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 12:27:39.370164 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.370128 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:27:39.370319 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.370307 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rs48x\"" Oct 31 12:27:39.370371 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.370349 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 12:27:39.377514 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.377491 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-67z7l"] Oct 31 12:27:39.377614 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.377599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.390238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.390205 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:39.397614 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.397593 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw"] Oct 31 12:27:39.397701 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.397682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.451110 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.451090 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:27:39.451191 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.451139 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 12:27:39.451191 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.451153 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 12:27:39.451191 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.451170 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6dgmh\"" Oct 31 12:27:39.451191 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.451170 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 12:27:39.451355 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.451241 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9c5q9\"" Oct 31 12:27:39.464539 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.464516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a95a2e68-c596-4c2e-b1bd-20d31bb31919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.464638 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.464546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.464638 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.464576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7qm\" (UniqueName: \"kubernetes.io/projected/a95a2e68-c596-4c2e-b1bd-20d31bb31919-kube-api-access-2t7qm\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.464723 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.464640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.469384 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.469369 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 12:27:39.469482 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.469452 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 12:27:39.519113 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.519095 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:39.565819 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.565796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.565904 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.565827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-accelerators-collector-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.565904 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.565847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c0a973f-9d43-4bec-a134-eda7d2b7a052-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.565981 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.565965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-wtmp\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.566015 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.565996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.566055 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxnw6\" (UniqueName: \"kubernetes.io/projected/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-api-access-wxnw6\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.566102 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.566132 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.566162 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-sys\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.566216 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-tls\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.566259 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.566339 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4f5z\" (UniqueName: \"kubernetes.io/projected/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-kube-api-access-z4f5z\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.566396 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.566488 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-textfile\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.566545 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-root\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.566545 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-metrics-client-ca\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.566632 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a95a2e68-c596-4c2e-b1bd-20d31bb31919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.566632 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7qm\" (UniqueName: \"kubernetes.io/projected/a95a2e68-c596-4c2e-b1bd-20d31bb31919-kube-api-access-2t7qm\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.566709 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.566645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.567214 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.567198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a95a2e68-c596-4c2e-b1bd-20d31bb31919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.568937 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.568918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.569080 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.569059 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.578813 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.578791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7qm\" (UniqueName: \"kubernetes.io/projected/a95a2e68-c596-4c2e-b1bd-20d31bb31919-kube-api-access-2t7qm\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.661602 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.661559 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:39.667520 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.667567 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-textfile\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667567 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-root\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667639 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-metrics-client-ca\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667639 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667639 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-root\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667770 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-accelerators-collector-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667770 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c0a973f-9d43-4bec-a134-eda7d2b7a052-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.667770 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-wtmp\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667770 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.667957 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnw6\" (UniqueName: \"kubernetes.io/projected/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-api-access-wxnw6\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.667957 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.667957 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-sys\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667957 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-tls\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667957 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.667957 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4f5z\" (UniqueName: \"kubernetes.io/projected/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-kube-api-access-z4f5z\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.667957 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667914 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-textfile\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.668311 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.667981 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-sys\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.668311 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.668104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c0a973f-9d43-4bec-a134-eda7d2b7a052-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.668311 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.668218 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-metrics-client-ca\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.668311 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.668274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.668517 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.668384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-wtmp\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.668517 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.668444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-accelerators-collector-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.668750 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.668728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.670220 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.670160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.670655 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.670634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-tls\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.670746 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.670653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.671093 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.671068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.680666 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.680649 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnw6\" (UniqueName: \"kubernetes.io/projected/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-api-access-wxnw6\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.681226 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.681205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4f5z\" (UniqueName: \"kubernetes.io/projected/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-kube-api-access-z4f5z\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.716746 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.716728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:27:39.788851 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.788828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:27:39.790871 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.790843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:39.809112 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.809085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:27:39.811275 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:39.811253 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fe0d0c_038c_4ac2_bbf3_fc5881fe3105.slice/crio-d3522e1a2df5c37983867d0146f13001a3dcbf177d6d255db4f1c587270bb701 WatchSource:0}: Error finding container d3522e1a2df5c37983867d0146f13001a3dcbf177d6d255db4f1c587270bb701: Status 404 returned error can't find the container with id d3522e1a2df5c37983867d0146f13001a3dcbf177d6d255db4f1c587270bb701 Oct 31 12:27:39.872834 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.872812 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57dbb8df47-wq5vh"] Oct 31 12:27:39.908000 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.906640 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57dbb8df47-wq5vh"] Oct 31 12:27:39.908000 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.906762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.911314 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.911927 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9nc4h\"" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.912085 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.912163 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.912212 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.912282 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.912308 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 12:27:39.913789 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.912407 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 12:27:39.977172 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:39.977142 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:40.020580 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.020557 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl"] Oct 31 12:27:40.023757 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:40.023734 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95a2e68_c596_4c2e_b1bd_20d31bb31919.slice/crio-53f674e2f0d19c82af83f0eb57c86faaefdf5ded97fb11ee9e0cd179cb56ebaf WatchSource:0}: Error finding container 53f674e2f0d19c82af83f0eb57c86faaefdf5ded97fb11ee9e0cd179cb56ebaf: Status 404 returned error can't find the container with id 53f674e2f0d19c82af83f0eb57c86faaefdf5ded97fb11ee9e0cd179cb56ebaf Oct 31 12:27:40.070101 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.070080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.070166 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.070111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.070224 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.070211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.070265 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.070232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.070265 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.070254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.070326 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.070268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.070367 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.070351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.123229 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.123212 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw"] Oct 31 12:27:40.127564 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:40.127535 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0a973f_9d43_4bec_a134_eda7d2b7a052.slice/crio-59359b45872d4afd13eeec5f538782c767ad10174592d8bccaffe1799987ed97 WatchSource:0}: Error finding container 59359b45872d4afd13eeec5f538782c767ad10174592d8bccaffe1799987ed97: Status 404 returned error can't find the container with id 59359b45872d4afd13eeec5f538782c767ad10174592d8bccaffe1799987ed97 Oct 31 12:27:40.171219 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.171297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.171297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.171297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.171297 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.171525 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.171525 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.171913 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.171889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.172217 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.172196 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.172278 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.172228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.172278 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.172247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.174533 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.174399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.174605 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.174430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.185603 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.185582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.248615 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.248594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:40.257569 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.257535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67z7l" event={"ID":"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105","Type":"ContainerStarted","Data":"d3522e1a2df5c37983867d0146f13001a3dcbf177d6d255db4f1c587270bb701"} Oct 31 12:27:40.258986 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.258964 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" event={"ID":"7c0a973f-9d43-4bec-a134-eda7d2b7a052","Type":"ContainerStarted","Data":"59359b45872d4afd13eeec5f538782c767ad10174592d8bccaffe1799987ed97"} Oct 31 12:27:40.260649 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.260628 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" event={"ID":"a95a2e68-c596-4c2e-b1bd-20d31bb31919","Type":"ContainerStarted","Data":"f41ba9ad78336d78e9af470567eaae937e7d4f8544ccd91826d4b7bee33759e1"} Oct 31 12:27:40.260716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.260670 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" event={"ID":"a95a2e68-c596-4c2e-b1bd-20d31bb31919","Type":"ContainerStarted","Data":"b2cd253f4b8aca072754a282127200fd1a990852090fa7ddf2debff118dec7be"} Oct 31 12:27:40.260716 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.260689 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" event={"ID":"a95a2e68-c596-4c2e-b1bd-20d31bb31919","Type":"ContainerStarted","Data":"53f674e2f0d19c82af83f0eb57c86faaefdf5ded97fb11ee9e0cd179cb56ebaf"} Oct 31 12:27:40.377835 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.377795 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:40.428509 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.428427 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:27:40.444283 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.444263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.449963 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.449943 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:27:40.451305 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451280 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 12:27:40.451415 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451325 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mjk8n\"" Oct 31 12:27:40.451415 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451363 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 12:27:40.451415 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451403 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 12:27:40.451616 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451436 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 12:27:40.451616 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451447 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 12:27:40.451616 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451511 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 12:27:40.451616 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451544 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 12:27:40.451616 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451563 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 12:27:40.451616 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.451596 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 12:27:40.543890 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.543820 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57dbb8df47-wq5vh"] Oct 31 12:27:40.546438 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.546410 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:40.546644 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:40.546623 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabb2014_99e4_4ae6_bf6a_ec7734ef86d0.slice/crio-5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0 WatchSource:0}: Error finding container 5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0: Status 404 returned error can't find the container with id 5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0 Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.573743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.573795 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.573821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.573862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.573937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79d28\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.582021 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.574489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675091 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675191 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675191 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675191 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79d28\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675712 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.675712 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.676142 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.675977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.676274 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.676250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.677574 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.677456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.678209 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.678188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.679812 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.679765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.680220 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.680203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.680992 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.680967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.681099 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.681076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.681391 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.681367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.681490 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.681417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.681835 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.681817 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.682679 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.682659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.682747 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.682680 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:40.682779 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.682746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.690445 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.690426 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79d28\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:40.836588 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:40.836556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:27:41.139288 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.139256 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:27:41.265844 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.265812 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"05afc9f1f06d7ed71042fbdbcc30ea7c79e1e53d369ff360e68405238fd537d3"} Oct 31 12:27:41.267616 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.267553 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dbb8df47-wq5vh" event={"ID":"babb2014-99e4-4ae6-bf6a-ec7734ef86d0","Type":"ContainerStarted","Data":"5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0"} Oct 31 12:27:41.427008 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.426900 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-577cf77cd5-p495x"] Oct 31 12:27:41.435956 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.435925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.443342 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.443517 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.443568 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.444140 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.444806 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hntd5\"" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.444850 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.444895 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b7krqr7laupp8\"" Oct 31 12:27:41.445759 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.445483 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-577cf77cd5-p495x"] Oct 31 12:27:41.584382 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584559 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584559 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2e7f16d-ec14-40bd-908f-272016e1464f-tmp\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584559 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-grpc-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584559 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbr8s\" (UniqueName: \"kubernetes.io/projected/d2e7f16d-ec14-40bd-908f-272016e1464f-kube-api-access-jbr8s\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584787 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584787 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584787 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e7f16d-ec14-40bd-908f-272016e1464f-metrics-client-ca\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.584787 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.584695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686221 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.685953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686412 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686507 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2e7f16d-ec14-40bd-908f-272016e1464f-tmp\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686573 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-grpc-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686632 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbr8s\" (UniqueName: \"kubernetes.io/projected/d2e7f16d-ec14-40bd-908f-272016e1464f-kube-api-access-jbr8s\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686690 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686831 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686894 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e7f16d-ec14-40bd-908f-272016e1464f-metrics-client-ca\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.686947 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.686915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.689806 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.689752 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2e7f16d-ec14-40bd-908f-272016e1464f-tmp\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.689993 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.689920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e7f16d-ec14-40bd-908f-272016e1464f-metrics-client-ca\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.691451 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.691361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-grpc-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.692308 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.692270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.692524 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.692506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.692935 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.692918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.694844 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.694804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.695521 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.695503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.702505 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.702482 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbr8s\" (UniqueName: \"kubernetes.io/projected/d2e7f16d-ec14-40bd-908f-272016e1464f-kube-api-access-jbr8s\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:41.764264 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:41.764101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:42.063524 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.063497 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-577cf77cd5-p495x"] Oct 31 12:27:42.273370 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.273288 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" event={"ID":"a95a2e68-c596-4c2e-b1bd-20d31bb31919","Type":"ContainerStarted","Data":"8a7b2c1be0fc6687b4b675292074f213a340f82e35645bca52b8408c68988e4c"} Oct 31 12:27:42.277162 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.276557 2578 generic.go:356] "Generic (PLEG): container finished" podID="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" containerID="6b77e7c2782f2ae913e1ba52e814ce6096cbcc9afda05fe0966f058465746cf2" exitCode=0 Oct 31 12:27:42.277162 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.276660 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67z7l" event={"ID":"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105","Type":"ContainerDied","Data":"6b77e7c2782f2ae913e1ba52e814ce6096cbcc9afda05fe0966f058465746cf2"} Oct 31 12:27:42.281196 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.281088 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6x24" event={"ID":"2726bde3-96ae-4717-8732-f7e792a0ddfb","Type":"ContainerStarted","Data":"cc4c7c8cae745dd3f5eaeece62cdefbe5ae91aa9c848311ae6d3fa58ce63fee8"} Oct 31 12:27:42.287579 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.287551 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" event={"ID":"d2e7f16d-ec14-40bd-908f-272016e1464f","Type":"ContainerStarted","Data":"a08818c28770ab580328f490ea04deab9b6644b8995a84faedf6e482d70b3bd5"} Oct 31 12:27:42.324035 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.323404 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" podStartSLOduration=1.550843065 podStartE2EDuration="3.323392271s" totalImagesPullingTime="1.772549206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:42.300526711 +0000 UTC m=+45.803825735" watchObservedRunningTime="2026-10-31 12:27:42.323392271 +0000 UTC m=+45.826691297" Oct 31 12:27:42.345962 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.345915 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b6x24" podStartSLOduration=2.460701542 podStartE2EDuration="6.345896713s" totalImagesPullingTime="3.885195171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:27:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:42.344583944 +0000 UTC m=+45.847882972" watchObservedRunningTime="2026-10-31 12:27:42.345896713 +0000 UTC m=+45.849195742" Oct 31 12:27:42.387232 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.387030 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:42.534712 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.534599 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:42.679687 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:42.679658 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:43.125130 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.125096 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:43.257244 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.257219 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:43.295928 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.295891 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" event={"ID":"7c0a973f-9d43-4bec-a134-eda7d2b7a052","Type":"ContainerStarted","Data":"5386cef1e6664dc580dbba418ce33ed5bcaac87278dbaf2d148e80938713fda4"} Oct 31 12:27:43.296380 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.295966 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" event={"ID":"7c0a973f-9d43-4bec-a134-eda7d2b7a052","Type":"ContainerStarted","Data":"9ee2c128f2f4bba1bc19e560a0a61335464137d3d339f4de3ee49352bb4964a7"} Oct 31 12:27:43.296380 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.295987 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" event={"ID":"7c0a973f-9d43-4bec-a134-eda7d2b7a052","Type":"ContainerStarted","Data":"a704d2ec3fa288f7ec3a9dee7be26c59fe12e38536f430900f778238ce529c1f"} Oct 31 12:27:43.298345 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.298295 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67z7l" event={"ID":"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105","Type":"ContainerStarted","Data":"1e42521c64112064e208434e457a291cdb89226396590640a1e89703be4e30f4"} Oct 31 12:27:43.298345 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.298340 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67z7l" event={"ID":"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105","Type":"ContainerStarted","Data":"74489832bd1878023332385e09349c2f76bc47ad89bc473d39da26c81d78fc94"} Oct 31 12:27:43.302491 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.300569 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937"} Oct 31 12:27:43.317871 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.317828 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" podStartSLOduration=1.7624462300000001 podStartE2EDuration="4.317816376s" totalImagesPullingTime="2.555370146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:43.31724176 +0000 UTC m=+46.820540788" watchObservedRunningTime="2026-10-31 12:27:43.317816376 +0000 UTC m=+46.821115403" Oct 31 12:27:43.341712 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:43.341665 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-67z7l" podStartSLOduration=2.439284165 podStartE2EDuration="4.341653203s" totalImagesPullingTime="1.823585942s" totalInitContainerRuntime="78.783096ms" isStatefulPod=true podCreationTimestamp="2026-10-31 12:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:43.34033352 +0000 UTC m=+46.843632547" watchObservedRunningTime="2026-10-31 12:27:43.341653203 +0000 UTC m=+46.844952233" Oct 31 12:27:44.165291 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.165258 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wqp22"] Oct 31 12:27:44.167805 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.167783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.172644 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.172619 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 12:27:44.172776 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.172691 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5kk9p\"" Oct 31 12:27:44.177876 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.177853 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wqp22"] Oct 31 12:27:44.211576 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.211388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kdf\" (UniqueName: \"kubernetes.io/projected/0d06e6e9-51c0-403a-8ea0-ecc9de005803-kube-api-access-p7kdf\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.211576 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.211440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d06e6e9-51c0-403a-8ea0-ecc9de005803-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.218843 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.218809 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:44.311855 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.311820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kdf\" (UniqueName: \"kubernetes.io/projected/0d06e6e9-51c0-403a-8ea0-ecc9de005803-kube-api-access-p7kdf\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.312215 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.311878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d06e6e9-51c0-403a-8ea0-ecc9de005803-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.312603 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.312574 2578 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937" exitCode=0 Oct 31 12:27:44.315838 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.313796 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937"} Oct 31 12:27:44.321216 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.321190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d06e6e9-51c0-403a-8ea0-ecc9de005803-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.328534 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.328512 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kdf\" (UniqueName: \"kubernetes.io/projected/0d06e6e9-51c0-403a-8ea0-ecc9de005803-kube-api-access-p7kdf\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.357316 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.357287 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:44.480238 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.480173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:44.763074 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:44.762994 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wqp22"] Oct 31 12:27:44.766928 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:27:44.766899 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d06e6e9_51c0_403a_8ea0_ecc9de005803.slice/crio-b4973ac2f763667fa8ca041361dc1eb087b5298b42d35c0496c943af4ea0629b WatchSource:0}: Error finding container b4973ac2f763667fa8ca041361dc1eb087b5298b42d35c0496c943af4ea0629b: Status 404 returned error can't find the container with id b4973ac2f763667fa8ca041361dc1eb087b5298b42d35c0496c943af4ea0629b Oct 31 12:27:45.318431 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.318405 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" event={"ID":"0d06e6e9-51c0-403a-8ea0-ecc9de005803","Type":"ContainerStarted","Data":"b4973ac2f763667fa8ca041361dc1eb087b5298b42d35c0496c943af4ea0629b"} Oct 31 12:27:45.630746 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.630715 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:27:45.635056 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.635006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.640934 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.640805 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 12:27:45.640934 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.640826 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 12:27:45.640934 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.640805 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 12:27:45.640934 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.640903 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 12:27:45.640934 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.640929 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 12:27:45.641266 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.641114 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 12:27:45.641266 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.641114 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 12:27:45.641266 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.641150 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tmrcz\"" Oct 31 12:27:45.641385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.641337 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 12:27:45.641470 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.641451 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 12:27:45.644003 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.643980 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:27:45.644121 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.644041 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 12:27:45.644414 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.644397 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fh91gmt3il7ar\"" Oct 31 12:27:45.645489 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.645454 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 12:27:45.645489 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.645478 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 12:27:45.649306 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.649276 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 12:27:45.649959 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.649943 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 12:27:45.653573 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.653552 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:27:45.693710 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.689893 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:45.725657 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.725745 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.725745 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.725856 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.725856 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.725856 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726007 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726007 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726007 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726007 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vx5z\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726007 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.725979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726179 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726179 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726179 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726179 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726179 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726345 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726345 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726345 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.726345 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.726264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.826919 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.826892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827011 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.826932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827011 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.826978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827011 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.826994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827103 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827103 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827103 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827103 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827245 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827245 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827245 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827245 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827351 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827351 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827351 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827351 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.827351 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.828118 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vx5z\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.828118 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.828118 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.828118 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.827798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.833528 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.829452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.833528 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.829959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.833528 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.830323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.833528 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.831280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.833528 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.831526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.837958 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.834616 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.837958 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.834741 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.837958 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.837621 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.838347 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.838325 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.838721 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.838695 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.838937 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.838708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.839370 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.839343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.840047 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.840020 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.840969 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.840945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.841237 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.841212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.841305 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.841253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.845320 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.845055 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.850932 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.850902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.851118 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.851094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vx5z\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:45.959845 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:45.959802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:46.250073 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.249927 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:27:46.327816 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.327348 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jlbzw" event={"ID":"c41fab2e-6e79-4fe1-a46e-996b4012efdf","Type":"ContainerStarted","Data":"98b1b52494f70e2432edf0f33a5885c08678a9f029fd37fc364c317c80d8ec25"} Oct 31 12:27:46.330858 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.330829 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dbb8df47-wq5vh" event={"ID":"babb2014-99e4-4ae6-bf6a-ec7734ef86d0","Type":"ContainerStarted","Data":"be2276cab2acc8cc4b99c70d4f52723b0135292202934d79608fc36ceb092915"} Oct 31 12:27:46.338567 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.338527 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" event={"ID":"d2e7f16d-ec14-40bd-908f-272016e1464f","Type":"ContainerStarted","Data":"174a5ebe7ab62b39c5ae170b0db72030463a3bb1df59a580fa558202b044929a"} Oct 31 12:27:46.338636 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.338594 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" event={"ID":"d2e7f16d-ec14-40bd-908f-272016e1464f","Type":"ContainerStarted","Data":"471e17d28c55dd89535340da516945ec001153354df693d01d9cd26769fcce34"} Oct 31 12:27:46.338701 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.338636 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" event={"ID":"d2e7f16d-ec14-40bd-908f-272016e1464f","Type":"ContainerStarted","Data":"b8345b7e3bdcfa2cf170eae893bcf43e7ca89c70ce74911c367b7b4382fa96d7"} Oct 31 12:27:46.342184 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.342162 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} Oct 31 12:27:46.342277 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.342226 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"70a4a90f0770abbc2d635d6b7a7987756c9c0d9357e3eec443ef0a7ea96a15a9"} Oct 31 12:27:46.344885 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.344858 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8"} Oct 31 12:27:46.390132 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.387512 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jlbzw" podStartSLOduration=2.658975216 podStartE2EDuration="10.387493178s" totalImagesPullingTime="7.728517962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:46.346688486 +0000 UTC m=+49.849987514" watchObservedRunningTime="2026-10-31 12:27:46.387493178 +0000 UTC m=+49.890792206" Oct 31 12:27:46.952015 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:46.951029 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:47.092956 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.092910 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:47.350438 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.350405 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4jx6k" event={"ID":"052e1a61-3220-4a91-83c5-a08bb829f358","Type":"ContainerStarted","Data":"98f5858165916d2bcb7277c33a812c5f26df51378027ecbf389d201f84fc99d4"} Oct 31 12:27:47.350835 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.350487 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4jx6k" event={"ID":"052e1a61-3220-4a91-83c5-a08bb829f358","Type":"ContainerStarted","Data":"c9149e85e4c67afa6be12f38a7b0a0c206561a44c0f0ae90a563e1d25fba9265"} Oct 31 12:27:47.350835 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.350519 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:47.353385 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.353361 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" event={"ID":"d2e7f16d-ec14-40bd-908f-272016e1464f","Type":"ContainerStarted","Data":"0adfddfba214b5da4ea73a7f107330ec0952495a3e5fe11ff25e7fa9ff8f61bd"} Oct 31 12:27:47.353525 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.353422 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" event={"ID":"d2e7f16d-ec14-40bd-908f-272016e1464f","Type":"ContainerStarted","Data":"d2018e6730b730e948292fba57b37dce1fb2297fa0c1779e3fe3340e8de678a3"} Oct 31 12:27:47.353525 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.353453 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" event={"ID":"d2e7f16d-ec14-40bd-908f-272016e1464f","Type":"ContainerStarted","Data":"5ecd0f430ec5c71aaf480b5b45867711fd79fbb97e3bad0986f63428de7d7e4d"} Oct 31 12:27:47.353525 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.353509 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:47.354925 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.354902 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" event={"ID":"0d06e6e9-51c0-403a-8ea0-ecc9de005803","Type":"ContainerStarted","Data":"97fd2a77ede58c2e9df96711bc959c98b47c040352dfccdc9951374900bf200c"} Oct 31 12:27:47.354993 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.354949 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:47.356597 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.356575 2578 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" exitCode=0 Oct 31 12:27:47.356694 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.356675 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} Oct 31 12:27:47.360490 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.360454 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb"} Oct 31 12:27:47.360574 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.360537 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb"} Oct 31 12:27:47.360631 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.360581 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec"} Oct 31 12:27:47.360631 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.360620 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4"} Oct 31 12:27:47.360736 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.360660 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerStarted","Data":"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831"} Oct 31 12:27:47.361060 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.361029 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:27:47.365914 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.365871 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:47.371296 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.371256 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57dbb8df47-wq5vh" podStartSLOduration=3.6551656169999998 podStartE2EDuration="8.371245849s" totalImagesPullingTime="4.716080232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:46.410283956 +0000 UTC m=+49.913582987" watchObservedRunningTime="2026-10-31 12:27:47.371245849 +0000 UTC m=+50.874544876" Oct 31 12:27:47.371629 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.371573 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-4jx6k" podStartSLOduration=2.053684626 podStartE2EDuration="11.371566712s" totalImagesPullingTime="9.317882086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:47.370731565 +0000 UTC m=+50.874030594" watchObservedRunningTime="2026-10-31 12:27:47.371566712 +0000 UTC m=+50.874865738" Oct 31 12:27:47.401698 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.401656 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" podStartSLOduration=1.811884571 podStartE2EDuration="6.401646113s" totalImagesPullingTime="4.589761542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:47.39952598 +0000 UTC m=+50.902825008" watchObservedRunningTime="2026-10-31 12:27:47.401646113 +0000 UTC m=+50.904945140" Oct 31 12:27:47.421274 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.421241 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" podStartSLOduration=1.421668688 podStartE2EDuration="3.421229194s" totalImagesPullingTime="1.999560506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:47.420399835 +0000 UTC m=+50.923698863" watchObservedRunningTime="2026-10-31 12:27:47.421229194 +0000 UTC m=+50.924528223" Oct 31 12:27:47.487958 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.487920 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.007566923 podStartE2EDuration="7.487906606s" totalImagesPullingTime="4.46296326s" totalInitContainerRuntime="17.376423ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:47.486852379 +0000 UTC m=+50.990151406" watchObservedRunningTime="2026-10-31 12:27:47.487906606 +0000 UTC m=+50.991205635" Oct 31 12:27:47.506889 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.506869 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:47.757898 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.757805 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:47.892793 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:47.892766 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:48.800353 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:48.800278 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:48.938642 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:48.938614 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:50.248801 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:50.248768 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:50.249214 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:50.248832 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:50.254080 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:50.254060 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:50.375763 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:50.375736 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:27:51.382551 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:51.382514 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} Oct 31 12:27:51.382901 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:51.382609 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} Oct 31 12:27:51.382901 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:51.382652 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} Oct 31 12:27:51.911887 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:51.911856 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:52.041903 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:52.041874 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:52.400158 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:52.400124 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:52.404284 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:52.404253 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} Oct 31 12:27:52.404417 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:52.404358 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} Oct 31 12:27:52.404417 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:52.404399 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerStarted","Data":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} Oct 31 12:27:52.449012 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:52.448957 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.665479043 podStartE2EDuration="7.448937569s" totalImagesPullingTime="3.761269384s" totalInitContainerRuntime="22.189142ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:52.445664174 +0000 UTC m=+55.948963203" watchObservedRunningTime="2026-10-31 12:27:52.448937569 +0000 UTC m=+55.952236597" Oct 31 12:27:52.534406 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:52.534363 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:53.370412 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:53.370387 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:27:53.768584 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:53.768507 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:53.912327 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:53.912299 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:54.058696 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:54.058607 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:55.960663 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:55.960626 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:27:56.232141 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:56.232056 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:27:56.460534 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:56.460502 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:56.626573 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:56.626549 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:57.367163 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:57.367132 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4jx6k" Oct 31 12:27:57.599516 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:57.599482 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:27:57.729881 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:27:57.729814 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:02.677830 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.677789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:28:02.681055 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.681034 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:28:02.690718 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.690685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:28:02.778903 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.778876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:02.782093 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.782076 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:28:02.790956 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.790935 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:02.880115 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.880085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:28:02.883080 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.883065 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:28:02.893109 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.893087 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:28:02.903227 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.903206 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:28:02.929147 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.929097 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\"" Oct 31 12:28:02.937192 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.937177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:28:02.943437 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.943424 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\"" Oct 31 12:28:02.950483 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:02.950454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:03.190377 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.190322 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\"" Oct 31 12:28:03.193083 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.193055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:03.198095 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.198073 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:28:03.211391 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.211371 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kr4dj"] Oct 31 12:28:03.213350 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:28:03.213328 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a428dff_76d1_44aa_864a_2532953ad0ad.slice/crio-6c6a8c24ebc68aa823920ce33f478657d2b6b193a264d2606abe6198d8fe66db WatchSource:0}: Error finding container 6c6a8c24ebc68aa823920ce33f478657d2b6b193a264d2606abe6198d8fe66db: Status 404 returned error can't find the container with id 6c6a8c24ebc68aa823920ce33f478657d2b6b193a264d2606abe6198d8fe66db Oct 31 12:28:03.227779 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.227752 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7zbh"] Oct 31 12:28:03.231275 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:28:03.231250 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda166dd5f_2e0c_45a0_8dc6_692741a8e1b3.slice/crio-6ee0417534535f6be00ca2415e5591f4371166f1e1fdca80eb59b3157986c752 WatchSource:0}: Error finding container 6ee0417534535f6be00ca2415e5591f4371166f1e1fdca80eb59b3157986c752: Status 404 returned error can't find the container with id 6ee0417534535f6be00ca2415e5591f4371166f1e1fdca80eb59b3157986c752 Oct 31 12:28:03.321591 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.321560 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:03.448836 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.448766 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7zbh" event={"ID":"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3","Type":"ContainerStarted","Data":"6ee0417534535f6be00ca2415e5591f4371166f1e1fdca80eb59b3157986c752"} Oct 31 12:28:03.449981 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.449953 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kr4dj" event={"ID":"7a428dff-76d1-44aa-864a-2532953ad0ad","Type":"ContainerStarted","Data":"6c6a8c24ebc68aa823920ce33f478657d2b6b193a264d2606abe6198d8fe66db"} Oct 31 12:28:03.474848 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:03.474826 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcknp"] Oct 31 12:28:03.478112 ip-10-0-142-45 kubenswrapper[2578]: W1031 12:28:03.478086 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda632a023_665e_4567_b2fc_13b7c37d152d.slice/crio-38261aa230e7c9dfecbef1ad3e00e6747ff2cbfb43dafec7c7da608c7151da95 WatchSource:0}: Error finding container 38261aa230e7c9dfecbef1ad3e00e6747ff2cbfb43dafec7c7da608c7151da95: Status 404 returned error can't find the container with id 38261aa230e7c9dfecbef1ad3e00e6747ff2cbfb43dafec7c7da608c7151da95 Oct 31 12:28:04.317055 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:04.317019 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:04.450305 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:04.450265 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:04.455953 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:04.455847 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zcknp" event={"ID":"a632a023-665e-4567-b2fc-13b7c37d152d","Type":"ContainerStarted","Data":"38261aa230e7c9dfecbef1ad3e00e6747ff2cbfb43dafec7c7da608c7151da95"} Oct 31 12:28:04.624994 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:04.624964 2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57dbb8df47-wq5vh"] Oct 31 12:28:05.462571 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:05.462534 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kr4dj" event={"ID":"7a428dff-76d1-44aa-864a-2532953ad0ad","Type":"ContainerStarted","Data":"2f5e589aa581f1741588cea6b9f0c18b32205440bcbc954a013122ed5af6ca5d"} Oct 31 12:28:05.463003 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:05.462593 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kr4dj" event={"ID":"7a428dff-76d1-44aa-864a-2532953ad0ad","Type":"ContainerStarted","Data":"167036f0f6ad7f43a8503f60cfad906dc8799af873e37a0023b452887b5b715a"} Oct 31 12:28:05.482248 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:05.482194 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kr4dj" podStartSLOduration=67.098993753 podStartE2EDuration="1m8.482177656s" totalImagesPullingTime="1.383183903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:28:05.480886502 +0000 UTC m=+68.984185528" watchObservedRunningTime="2026-10-31 12:28:05.482177656 +0000 UTC m=+68.985476683" Oct 31 12:28:05.637250 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:05.637221 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:05.772680 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:05.772611 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:05.922504 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:05.922475 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:07.339853 ip-10-0-142-45 kubenswrapper[2578]: E1031 12:28:07.339822 2578 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:28:07.386845 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:07.386820 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:07.520547 ip-10-0-142-45 kubenswrapper[2578]: I1031 12:28:07.520516 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:09.113938 ip-10-0-142-45 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 12:28:09.135894 ip-10-0-142-45 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 12:28:09.136060 ip-10-0-142-45 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 12:28:09.136228 ip-10-0-142-45 systemd[1]: kubelet.service: Consumed 4.654s CPU time. Oct 31 12:28:09.145991 ip-10-0-142-45 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:28:09.212474 ip-10-0-142-45 kubenswrapper[6727]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:28:09.212899 ip-10-0-142-45 kubenswrapper[6727]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:28:09.212899 ip-10-0-142-45 kubenswrapper[6727]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:28:09.212899 ip-10-0-142-45 kubenswrapper[6727]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:28:09.214549 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214450 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.214549 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214548 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214553 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214559 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214562 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214566 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214569 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214572 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214575 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214577 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214580 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214583 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214585 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214588 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214591 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214594 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214598 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214602 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214605 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214616 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.214629 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214619 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214622 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214625 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214627 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214631 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214633 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214636 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214639 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214641 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214644 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214647 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214650 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214652 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214655 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214657 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214660 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214663 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214667 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214669 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.215155 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214673 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214675 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214678 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214681 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214683 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214686 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214688 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214691 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214693 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214696 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214699 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214701 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214704 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214707 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214710 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214713 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214715 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214719 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214721 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.215649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214724 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214726 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214729 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214732 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214735 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214737 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214740 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214742 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214744 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214747 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214750 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214753 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214756 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214759 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214761 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214764 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214767 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214769 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214772 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214775 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.216122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214778 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214781 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214783 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214786 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214788 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214791 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214793 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214796 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214798 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214801 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214804 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214807 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214809 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214812 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214814 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214817 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214820 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214822 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214825 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.216635 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214827 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214830 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214832 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214835 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214837 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214840 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214848 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214851 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.214853 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215642 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215649 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215652 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215655 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215658 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215661 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215663 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215666 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215672 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215675 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215679 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.217103 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215684 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215687 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215690 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215693 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215696 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215698 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215701 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215703 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215706 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215709 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215712 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215714 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215717 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215719 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215722 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215724 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215727 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215730 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215733 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215736 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.217603 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215738 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215741 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215743 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215746 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215749 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215751 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215754 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215756 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215759 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215765 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215767 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215770 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215773 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215775 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215778 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215782 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215785 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215787 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215790 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.218110 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215792 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215795 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215798 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215800 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215802 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215805 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215808 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215810 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215813 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215815 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215820 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215822 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215825 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215828 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215831 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215833 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215836 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215839 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215841 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215844 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.218599 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215846 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215849 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215852 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215855 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215857 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215860 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215862 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215865 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215867 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215870 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215873 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215875 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215878 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215881 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215884 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215887 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215889 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215892 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215894 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.219095 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215897 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215900 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215902 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215905 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215907 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215910 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215912 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215915 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215918 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215921 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215923 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215926 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215931 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215934 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215937 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215942 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.215945 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216385 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216390 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216394 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.219576 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216396 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216399 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216402 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216405 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216407 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216410 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216413 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216416 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216418 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216421 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216424 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216426 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216429 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216433 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216438 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216442 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216445 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216447 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216449 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.220105 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216453 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216455 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216458 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216476 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216480 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216484 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216488 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216492 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216497 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216500 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216503 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216506 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216509 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216511 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216514 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216517 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216520 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216522 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216525 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.220670 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216527 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216530 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216533 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216535 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216538 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216540 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216543 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216546 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216548 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216551 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216553 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216556 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216558 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216561 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216565 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216568 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216570 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216574 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216578 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.221434 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216582 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216585 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216589 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216592 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216594 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216597 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216600 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216602 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216605 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216608 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216611 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216613 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216616 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216619 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216621 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216624 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216626 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216629 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216631 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216634 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.221924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216637 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216639 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216642 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216644 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216647 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216650 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216652 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216655 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216658 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216661 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216664 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216666 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216669 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216671 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216675 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216677 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216680 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216682 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216685 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.222426 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216688 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216691 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216693 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216696 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216698 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216701 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.216703 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216790 6727 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216797 6727 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216803 6727 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216807 6727 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216811 6727 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216815 6727 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216821 6727 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216825 6727 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216828 6727 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216831 6727 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216835 6727 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216838 6727 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216841 6727 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216844 6727 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216865 6727 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216872 6727 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:28:09.222914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216875 6727 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216879 6727 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216883 6727 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216886 6727 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216889 6727 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216892 6727 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216895 6727 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216899 6727 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216902 6727 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216905 6727 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216908 6727 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216911 6727 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216914 6727 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216917 6727 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216920 6727 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216924 6727 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216928 6727 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216931 6727 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216934 6727 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216937 6727 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216940 6727 flags.go:64] FLAG: --enable-server="true" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216943 6727 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216947 6727 flags.go:64] FLAG: --event-burst="100" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216950 6727 flags.go:64] FLAG: --event-qps="50" Oct 31 12:28:09.223489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216953 6727 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216956 6727 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216959 6727 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216963 6727 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216966 6727 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216969 6727 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216972 6727 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216976 6727 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216979 6727 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216982 6727 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216985 6727 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216988 6727 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216991 6727 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216994 6727 flags.go:64] FLAG: --feature-gates="" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.216998 6727 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217001 6727 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217004 6727 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217007 6727 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217010 6727 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217013 6727 flags.go:64] FLAG: --help="false" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217016 6727 flags.go:64] FLAG: --hostname-override="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217019 6727 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217022 6727 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:28:09.224078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217025 6727 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217029 6727 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217032 6727 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217035 6727 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217038 6727 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217040 6727 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217043 6727 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217046 6727 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217049 6727 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217052 6727 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217055 6727 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217057 6727 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217060 6727 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217063 6727 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217066 6727 flags.go:64] FLAG: --lock-file="" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217070 6727 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217074 6727 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217077 6727 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217083 6727 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217086 6727 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217089 6727 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217092 6727 flags.go:64] FLAG: --logging-format="text" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217095 6727 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217098 6727 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:28:09.224643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217101 6727 flags.go:64] FLAG: --manifest-url="" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217104 6727 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217108 6727 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217112 6727 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217116 6727 flags.go:64] FLAG: --max-pods="110" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217119 6727 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217122 6727 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217125 6727 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217128 6727 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217131 6727 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217134 6727 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217137 6727 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217143 6727 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217146 6727 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217149 6727 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217152 6727 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217155 6727 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217158 6727 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217161 6727 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217164 6727 flags.go:64] FLAG: --port="10250" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217167 6727 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217170 6727 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0107acac4cee1cdd1" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217173 6727 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217178 6727 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217182 6727 flags.go:64] FLAG: --register-node="true" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217185 6727 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:28:09.225216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217189 6727 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217192 6727 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217196 6727 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217198 6727 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217202 6727 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217205 6727 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217208 6727 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217211 6727 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217215 6727 flags.go:64] FLAG: --runonce="false" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217218 6727 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217221 6727 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217224 6727 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217227 6727 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217230 6727 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217233 6727 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217236 6727 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217239 6727 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217241 6727 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217244 6727 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217247 6727 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217250 6727 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217253 6727 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217256 6727 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217259 6727 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217264 6727 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:28:09.225895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217267 6727 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217270 6727 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217274 6727 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217277 6727 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217281 6727 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217285 6727 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217288 6727 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217291 6727 flags.go:64] FLAG: --v="2" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217296 6727 flags.go:64] FLAG: --version="false" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217301 6727 flags.go:64] FLAG: --vmodule="" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217305 6727 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217308 6727 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217428 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217432 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217434 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217437 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217440 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217443 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217445 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217448 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217451 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217453 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.226531 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217456 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217458 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217475 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217479 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217483 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217487 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217491 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217496 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217500 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217503 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217506 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217508 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217511 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217514 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217518 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217523 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217525 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217528 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217531 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217533 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.227067 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217537 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217540 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217543 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217545 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217548 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217552 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217555 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217558 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217561 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217564 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217566 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217569 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217571 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217574 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217576 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217579 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217582 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217586 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217590 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.227679 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217593 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217596 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217598 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217601 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217604 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217607 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217609 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217614 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217617 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217620 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217623 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217626 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217628 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217631 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217634 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217637 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217640 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217642 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217645 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.228143 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217648 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217650 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217653 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217655 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217658 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217660 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217663 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217665 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217668 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217670 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217673 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217676 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217678 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217681 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217684 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217686 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217689 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217692 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217694 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.228946 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217697 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217700 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217704 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217707 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217709 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217712 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217714 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217717 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217719 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217728 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217731 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217734 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217736 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217739 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217742 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217744 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217747 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217749 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.229649 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217752 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.217778 6727 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217811 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217815 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217818 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217821 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217823 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217826 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217829 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217832 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217834 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217837 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217839 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217842 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217844 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217847 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.230186 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217852 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217855 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217857 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217860 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217863 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217866 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217868 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217871 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217874 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217877 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217879 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217882 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217884 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217887 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217889 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217892 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217894 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217897 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217900 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.230607 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217902 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217905 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217907 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217910 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217912 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217915 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217918 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217920 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217923 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217925 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217928 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217930 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217933 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217938 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217940 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217943 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217945 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217948 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217950 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217953 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.231068 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217956 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217958 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217961 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217964 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217967 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217969 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217971 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217974 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217977 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217981 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217984 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217986 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217988 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217991 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217994 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217996 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.217999 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218001 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218004 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218006 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.231705 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218009 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218011 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218014 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218016 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218019 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218030 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218034 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218036 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218039 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218042 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218045 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218048 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218050 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218053 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218055 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218058 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218061 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218064 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218066 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.232318 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218069 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218073 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218076 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218080 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218083 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218085 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218088 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218090 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218093 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218096 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218098 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218101 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218103 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.218108 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.222599 6727 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.222615 6727 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.222642 6727 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.222647 6727 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222705 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.232868 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222711 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222715 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222719 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222723 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222727 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222731 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222735 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222738 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222741 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222744 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222748 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222751 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222754 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222757 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222760 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222763 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222766 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222768 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222771 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.233338 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222774 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222776 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222779 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222782 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222785 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222787 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222790 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222793 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222796 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222799 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222801 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222804 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222807 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222809 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222812 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222815 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222818 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222820 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222823 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.233832 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222825 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222828 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222830 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222833 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222836 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222838 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222841 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222844 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222847 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222850 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222853 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222855 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222858 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222860 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222863 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222866 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222869 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222871 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222874 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222877 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.234297 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222880 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222882 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222885 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222888 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222891 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222893 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222896 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222898 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222901 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222903 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222906 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222908 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222911 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222913 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222916 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222919 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222922 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222924 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222927 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.234800 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222930 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222933 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222936 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222939 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222942 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222945 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222947 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222950 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222953 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222956 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222958 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222961 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222963 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222966 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222968 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222971 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222973 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222976 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222979 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222981 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.235331 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222984 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222986 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222989 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222991 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222994 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222996 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.222999 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223001 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.223027 6727 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223169 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223173 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223177 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223180 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223183 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223186 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223190 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.235823 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223195 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223198 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223202 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223205 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223207 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223210 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223212 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223215 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223217 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223220 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223223 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223225 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223228 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223230 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223233 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223236 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223239 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223241 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223244 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223246 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.236203 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223248 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223251 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223254 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223257 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223259 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223262 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223264 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223267 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223269 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223272 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223274 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223277 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223280 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223283 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223285 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223288 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223291 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223293 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223296 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223298 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.236738 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223301 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223304 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223307 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223310 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223313 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223315 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223318 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223320 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223323 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223325 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223328 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223330 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223333 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223336 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223339 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223341 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223344 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223346 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223349 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.237243 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223351 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223354 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223357 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223359 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223362 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223365 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223367 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223370 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223373 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223375 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223377 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223380 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223383 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223385 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223388 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223390 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223392 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223395 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223397 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223400 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.237729 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223402 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223405 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223408 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223410 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223413 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223415 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223418 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223421 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223423 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223426 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223428 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223431 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223434 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223438 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223441 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223443 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223445 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223448 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223451 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.238253 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223453 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.223498 6727 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223531 6727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223536 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223539 6727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223542 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223545 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223549 6727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223552 6727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223555 6727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223558 6727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223561 6727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223563 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223566 6727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223568 6727 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223571 6727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:28:09.238769 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223573 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223576 6727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223579 6727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223581 6727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223584 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223587 6727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223589 6727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223592 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223594 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223597 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223601 6727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223604 6727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223606 6727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223609 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223612 6727 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223614 6727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223618 6727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223620 6727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223623 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223625 6727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:28:09.239300 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223628 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223630 6727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223633 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223635 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223638 6727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223641 6727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223643 6727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223646 6727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223650 6727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223653 6727 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223656 6727 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223658 6727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223661 6727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223665 6727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223669 6727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223672 6727 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223675 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223677 6727 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223680 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:28:09.240114 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223683 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223685 6727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223688 6727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223692 6727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223695 6727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223698 6727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223700 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223703 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223705 6727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223708 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223711 6727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223714 6727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223716 6727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223719 6727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223721 6727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223724 6727 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223726 6727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223729 6727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223732 6727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:28:09.240631 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223734 6727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223737 6727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223741 6727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223744 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223746 6727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223749 6727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223752 6727 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223754 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223757 6727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223759 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223762 6727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223764 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223767 6727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223769 6727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223772 6727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223775 6727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223779 6727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223782 6727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223784 6727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223787 6727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:28:09.241159 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223790 6727 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223792 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223795 6727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223798 6727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223801 6727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223804 6727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223807 6727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223809 6727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223812 6727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223814 6727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223817 6727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223819 6727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223822 6727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:09.223824 6727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.223876 6727 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.224814 6727 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.224856 6727 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.225031 6727 server.go:1014] "Starting client certificate rotation" Oct 31 12:28:09.241677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.225179 6727 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.225240 6727 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:21:57 +0000 UTC" deadline="2028-04-11 01:07:36.765692147 +0000 UTC" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.225285 6727 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12660h39m27.540410892s" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.225556 6727 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.227499 6727 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.229324 6727 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.231456 6727 log.go:25] "Validated CRI v1 image API" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.232197 6727 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:28:09.242159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.234839 6727 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80b4c50b-55a8-4f1b-a3e1-91c614682cb8:/dev/nvme0n1p4 842b1835-9a5d-4357-9e49-bd18f54c6546:/dev/nvme0n1p3] Oct 31 12:28:09.242387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.234860 6727 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05afc9f1f06d7ed71042fbdbcc30ea7c79e1e53d369ff360e68405238fd537d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05afc9f1f06d7ed71042fbdbcc30ea7c79e1e53d369ff360e68405238fd537d3/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d436ac1dd286e85aa10a55cdbd5f2b547f728710da521dcf6b7476fb0db35d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d436ac1dd286e85aa10a55cdbd5f2b547f728710da521dcf6b7476fb0db35d8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38261aa230e7c9dfecbef1ad3e00e6747ff2cbfb43dafec7c7da608c7151da95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38261aa230e7c9dfecbef1ad3e00e6747ff2cbfb43dafec7c7da608c7151da95/userdata/shm major:0 minor:479 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c0e28ad18a3cf4138d38ee49042564bc6d06463c1742785f511280265b3cc14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c0e28ad18a3cf4138d38ee49042564bc6d06463c1742785f511280265b3cc14/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53f674e2f0d19c82af83f0eb57c86faaefdf5ded97fb11ee9e0cd179cb56ebaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53f674e2f0d19c82af83f0eb57c86faaefdf5ded97fb11ee9e0cd179cb56ebaf/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59359b45872d4afd13eeec5f538782c767ad10174592d8bccaffe1799987ed97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59359b45872d4afd13eeec5f538782c767ad10174592d8bccaffe1799987ed97/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c899e55c08191e87dc2980e9d90e93ed51866455f80ef6c79196ad68cfa65f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c899e55c08191e87dc2980e9d90e93ed51866455f80ef6c79196ad68cfa65f9/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c6a8c24ebc68aa823920ce33f478657d2b6b193a264d2606abe6198d8fe66db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c6a8c24ebc68aa823920ce33f478657d2b6b193a264d2606abe6198d8fe66db/userdata/shm major:0 minor:475 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ee0417534535f6be00ca2415e5591f4371166f1e1fdca80eb59b3157986c752/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ee0417534535f6be00ca2415e5591f4371166f1e1fdca80eb59b3157986c752/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70a4a90f0770abbc2d635d6b7a7987756c9c0d9357e3eec443ef0a7ea96a15a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70a4a90f0770abbc2d635d6b7a7987756c9c0d9357e3eec443ef0a7ea96a15a9/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82c724bb90c5850cdda5082faa02beba25df46a91866906e87b98d72f7f0fb83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82c724bb90c5850cdda5082faa02beba25df46a91866906e87b98d72f7f0fb83/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a08818c28770ab580328f490ea04deab9b6644b8995a84faedf6e482d70b3bd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a08818c28770ab580328f490ea04deab9b6644b8995a84faedf6e482d70b3bd5/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aca7a2bbb04671497f30cbb3751e89c0db6e7b379e60eb22f1854b16a8dacac0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aca7a2bbb04671497f30cbb3751e89c0db6e7b379e60eb22f1854b16a8dacac0/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3950d83734febb3fbfa39c130771e6be499b0cfe63820b169f74db1d5e37c98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3950d83734febb3fbfa39c130771e6be499b0cfe63820b169f74db1d5e37c98/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4973ac2f763667fa8ca041361dc1eb087b5298b42d35c0496c943af4ea0629b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4973ac2f763667fa8ca041361dc1eb087b5298b42d35c0496c943af4ea0629b/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c68c679ea306addeee1bd266a45a59452fb0ac79e7d0cb157e93aa62a29e0048/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c68c679ea306addeee1bd266a45a59452fb0ac79e7d0cb157e93aa62a29e0048/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c742a42beed1730c1859e9d6b22dadf86deb83a71a721a68b5cba1916362961f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c742a42beed1730c1859e9d6b22dadf86deb83a71a721a68b5cba1916362961f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3522e1a2df5c37983867d0146f13001a3dcbf177d6d255db4f1c587270bb701/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3522e1a2df5c37983867d0146f13001a3dcbf177d6d255db4f1c587270bb701/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3b15170a311240f888e8ec6647dec45c00df493eababc9b7b485e847c28cf22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3b15170a311240f888e8ec6647dec45c00df493eababc9b7b485e847c28cf22/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e15db8cdab7ba7373a055383a82face00539d0e5cc00432bad7c3c623fcf5ad8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e15db8cdab7ba7373a055383a82face00539d0e5cc00432bad7c3c623fcf5ad8/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9a67f7b03a671338555684404a3efa934cefa0e6e14f71c12515c7b624bfded/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9a67f7b03a671338555684404a3efa934cefa0e6e14f71c12515c7b624bfded/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/052e1a61-3220-4a91-83c5-a08bb829f358/volumes/kubernetes.io~projected/kube-api-access-qdpk5:{mountpoint:/var/lib/kubelet/pods/052e1a61-3220-4a91-83c5-a08bb829f358/volumes/kubernetes.io~projected/kube-api-access-qdpk5 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/052e1a61-3220-4a91-83c5-a08bb829f358/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/052e1a61-3220-4a91-83c5-a08bb829f358/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d06e6e9-51c0-403a-8ea0-ecc9de005803/volumes/kubernetes.io~projected/kube-api-access-p7kdf:{mountpoint:/var/lib/kubelet/pods/0d06e6e9-51c0-403a-8ea0-ecc9de005803/volumes/kubernetes.io~projected/kube-api-access-p7kdf major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d06e6e9-51c0-403a-8ea0-ecc9de005803/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/0d06e6e9-51c0-403a-8ea0-ecc9de005803/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/config-reloader/2 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/prometheus/15 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~empty-dir/config-out major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~projected/kube-api-access-2vx5z:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~projected/kube-api-access-2vx5z major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~projected/tls-assets major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/config major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/web-config major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2726bde3-96ae-4717-8732-f7e792a0ddfb/volumes/kubernetes.io~projected/kube-api-access-p7tf9:{mountpoint:/var/lib/kubelet/pods/2726bde3-96ae-4717-8732-f7e792a0ddfb/volumes/kubernetes.io~projected/kube-api-access-p7tf9 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2726bde3-96ae-4717-8732-f7e792a0ddfb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2726bde3-96ae-4717-8732-f7e792a0ddfb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32afbb71-9024-4d12-a27a-994892cf38e4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/32afbb71-9024-4d12-a27a-994892cf38e4/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/alertmanager/9 major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/config-reloader/6 major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~empty-dir/config-out major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~projected/kube-api-access-79d28:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~projected/kube-api-access-79d28 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~projected/tls-assets major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/config-volume major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/web-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d278316-af3e-4b36-8505-a5d464016e85/volumes/kubernetes.io~projected/kube-api-access-zvgcz:{mountpoint:/var/lib/kubelet/pods/6d278316-af3e-4b36-8505-a5d464016e85/volumes/kubernetes.io~projected/kube-api-access-zvgcz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a428dff-76d1-44aa-864a-2532953ad0ad/volumes/kubernetes.io~projected/kube-api-access-djjrr:{mountpoint:/var/lib/kubelet/pods/7a428dff-76d1-44aa-864a-2532953ad0ad/volumes/kubernetes.io~projected/kube-api-access-djjrr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a428dff-76d1-44aa-864a-2532953ad0ad/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7a428dff-76d1-44aa-864a-2532953ad0ad/volumes/kubernetes.io~secret/metrics-certs major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~projected/kube-api-access-wxnw6:{mountpoint:/var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~projected/kube-api-access-wxnw6 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82ecc097-1c88-4f8c-928c-708b9a628a72/volumes/kubernetes.io~projected/kube-api-access-z79d2:{mountpoint:/var/lib/kubelet/pods/82ecc097-1c88-4f8c-928c-708b9a628a72/volumes/kubernetes.io~projected/kube-api-access-z79d2 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fc0c5ec-9758-40ff-9990-ab568b8d02dd/volumes/kubernetes.io~projected/kube-api-access-8vzgv:{mountpoint:/var/lib/kubelet/pods/8fc0c5ec-9758-40ff-9990-ab568b8d02dd/volumes/kubernetes.io~projected/kube-api-access-8vzgv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~projected/kube-api-access-qcsz9:{mountpoint:/var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~projected/kube-api-access-qcsz9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:473 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a632a023-665e-4567-b2fc-13b7c37d152d/volumes/kubernetes.io~projected/kube-api-access-pw92g:{mountpoint:/var/lib/kubelet/pods/a632a023-665e-4567-b2fc-13b7c37d152d/volumes/kubernetes.io~projected/kube-api-access-pw92g major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~projected/kube-api-access-2t7qm:{mountpoint:/var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~projected/kube-api-access-2t7qm major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad2b3ee2-f439-49ca-a106-63e231dc232f/volumes/kubernetes.io~projected/kube-api-access-7qglj:{mountpoint:/var/lib/kubelet/pods/ad2b3ee2-f439-49ca-a106-63e231dc232f/volumes/kubernetes.io~projected/kube-api-access-7qglj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1925581-bb98-4bd2-8c66-5cafa370dfc9/volumes/kubernetes.io~projected/kube-api-access-r2dbt:{mountpoint:/var/lib/kubelet/pods/b1925581-bb98-4bd2-8c66-5cafa370dfc9/volumes/kubernetes.io~projected/kube-api-access-r2dbt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b626d907-7b4b-4437-9e69-a82333ebf268/volumes/kubernetes.io~projected/kube-api-access-j6w2l:{mountpoint:/var/lib/kubelet/pods/b626d907-7b4b-4437-9e69-a82333ebf268/volumes/kubernetes.io~projected/kube-api-access-j6w2l major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~projected/kube-api-access-ckfkv:{mountpoint:/var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~projected/kube-api-access-ckfkv major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c41fab2e-6e79-4fe1-a46e-996b4012efdf/volumes/kubernetes.io~projected/kube-api-access-wk9c6:{mountpoint:/var/lib/kubelet/pods/c41fab2e-6e79-4fe1-a46e-996b4012efdf/volumes/kubernetes.io~projected/kube-api-access-wk9c6 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c41fab2e-6e79-4fe1-a46e-996b4012efdf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c41fab2e-6e79-4fe1-a46e-996b4012efdf/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~projected/kube-api-access-jbr8s:{mountpoint:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~projected/kube-api-access-jbr8s major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~projected/kube-api-access-z4f5z:{mountpoint:/var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~projected/kube-api-access-z4f5z major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volumes/kubernetes.io~projected/kube-api-access-ppm8b:{mountpoint:/var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volumes/kubernetes.io~projected/kube-api-access-ppm8b major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a71cbc421184123d338e7251fc7744f0cb28ab7c6adfa414161ac9b5f1571c1a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1f013a9e7237600457b4b51e0d26b8709acc39419194dfc9208332b3a94de279/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f3f078d364d02f767f825e90799b202fc23f337e39fe6b13a16842d4de5ae9e6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f2eac4af40e832cc009ed9a09cccea059feef75eb940ba9d691644ea9b34fb2e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0e4dfea5744a34c08d4fdf2ae61edc84e198856e3d3330d5d52c97d682df2c39/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2afde201262532b0ece2ca19022f63640529af2aa5543aea110d89f9c5255637/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/16949f4259b7eecbf869c3d68c43e011aa21f6ab76833e3794b3f889910fcd63/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2d4ad6e438bbe96216ac1cf38e84eedb303cc2b4dc585a9f7d7b7988eceed6f9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f3fa3b37d4e13df4ea3829169fc5c654cc4a515831015f28958bd020a82d74fd/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9c1b28151b9f22be50081c9a6ccac1317e7ae89f269d7227f7282a6514c9eff9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/09c7dd4360e6b88c5888a7b4adca274e81c1e7cc75b2cab197cb42a0c54e01ba/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/6dc4e4a1d89518a51e60b785ea8b3f6f0972e3c9a328be0aa48c62fb5230eb1e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a3e3302b1e7da484aedc77943861962670a3d7680860896481e4aee938d389c5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/95346fa2ccd25627d91570c5426b4d74823a1d75fe13d1748c641570f56ed998/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/82f076c983b25f38337351cdba50824f3cfb5a58fa1c70017950c1f1f8c819c2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/abe6d3d55cb05963ebc573adb196997985c9f8d3a4f8872174a40336a2e126ea/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4691dbe2969f9cacea5cbc9937ec1e27f140c17ff34563837ebdcce6bddc623d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4843413a2a55a16237463baae57827e96501c1cc9c789f24670ed12079c4f943/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/84e888e5be4f1276d95ce104f6665fbe80102329b75b199cab626e8ed10e7b81/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/81c9111e5425ffe416b418970b367916070b6d60973ac691649fe7309f34bc42/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/546b2c59651f10b35cdda2e180b3734e6795e05dd2b2aab8c3488539b2a9e02b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/deded51fd31da1be21a861160ae9fdf6cb6c78504cb064d40307feb50b1e1069/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/7b17ec203b1ccb0e06fe54071be015a119238d1d9cf0e4b5523c130dc50fd5c8/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/fd2dcb2312f7d09b8cf789d2f4369761db97ff1e53d56788ca3a95217ee2dac1/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/908a63a7dd433edc3cfd9cc0da17034b41af6295048186330bdece3998aed08d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/cf605ded2b4c6935bceb4b2245e9770a1f05c182bdf1a6b193ee6063e3e7360e/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/bb9dd19d29a816c1c1ebd59917fdffba9a26d319ef8433cc3c6ecb6ed79618a9/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/6c44002baefc82a3196b9bec879b5cba81f87242c46e41eb76dfeb5887ad0de3/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/4893ee0427d237c3df4fddee502cb1cca23b039d1ca3e423f1da92e85ab55aea/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/d1a15b9720ebf35cd801a58502e3202574da7c7403472851fbe4ee1e7c37a296/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/574de26f35d5d32961c22b7fb96b69d47044afd11b6f55ac2477e8795c4fc91d/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/c0dff25400eaedf6f1c9b58a7a2d2eed096a00488a82ac0482d7de00653a6f74/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/c5aedadcd883c6550dbf37e9f642fbb961a8956567e75f5dac177a45a062725f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/1d8fdff66e824d4d1aac32500027d14d85baf957ac4c7db1646088d4490e4db6/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/a3108bdb9e310f81e94d942aac3de18609dc8cd4fba511cff8b60ef012d30ffd/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/4fdafcb785952c58691863d984bd7d1607bfc39ae9a1477a228ef9e88872db83/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/0ad5a91adf72e99b08f53915fd06b3bcdbe43df4ebc82f8c5fead7afb020e83f/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/8772fe87fce5df3a61e73da8a9785488e19ae7df5e5242d54f9fd2267b3b7300/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/13f73ef7a8effae32932bf00ab61a0516de03efc9023f026beca2eb85cf2fffb/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/4be2982e5dfcdce9ad5b400362214de74d2ec279b65b163f97c6366932206497/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/e4f29488a80aaefd0d914a2cad0a4f1a9e3e6e4a6e5b31b7f8250daf304470b0/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/e96a54101438567266245b5bb7cda182c61ae069916d9b017f618ae66d46ec80/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/9154a00a2202ffb992a07a6f04ccd9fd6ff496f5dbb451309d33ad0c447ef30f/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/24c14101613838f2b1574343c6e4af8cba84dff8591eb30716d29604de672447/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/914a8c6c73cf3cc7e6b5d20f111b100245d91c819721fd49ca005b3d87c3128d/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/de95718347f0c2386507e07fa2bf1bb4c4fb9b91ad6997791cda7bb5b8747e80/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/bcc899c3f0444f42fff66376f5456229c43de3313e046fdcae0ca2674b9f7a46/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/03fbe96a75c02fd1ac57fc8b8efd9623eeb661aad5959b8ab707d0ab0ae60eba/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/eee710de986a0b93bbfadea0405af44bd755f328dbabf52d72aea57af810a944/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/69ea894061e023224b868a9ac952481668ee291de57faf56581e2898571b0223/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/750fa3e09749e0955e98646816397a13a63b1092a33cd4f5598cc971b9491964/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/736c9edb6d05a5cf16314e32fbf07abfb5a70a9974cf6bd0e3d27b1e29b37915/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/f4b33e5998293eb37dad86bede06cbe5f096f888ee5f7a30961999bb17ca4558/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/9c8f571558e2d848b2259b7159d26c29d3a2335cdd0d12d66b250b3b2f9c488e/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/4c7bb9ee3288e248883f04396efb633d728f8451d47ed64640ae7303ce9123c1/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/6d6d61965b0df9e1df52a2db590969b7f15e2a800f6f543b530d946ac419dd44/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/fc3e4362ae169e43a039ad719c328b9fca4a15e29a245a85dce5d890a0306339/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/1cb77c1c73844e46d3b69dcd7e2208095c219bb536f10853434263f724aa498b/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/e8c41310d794269c538db0de7ccb73b717ec32de00af774006f55d2491a16f4e/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/ca6e69ef4ee4fc4ce7d61a18ccdfc59e2c6c34481ccb509ecc60be09af56112e/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/f6f63389248b5f5b9dcc52c536a78e200bd60bd38a492641e2f351bb12819fab/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/7df33e1a77bdde5c2be318e9716abb4904176dcbdedc908ea301a9a3fc6c0899/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/957da66adb523dc1f9c88e87af3479f57666e98c44f56c4a41d1b5ae5a26a7fe/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/4bcc9e9c761507a00f5f1c7f2d7eba0f6a85c177ef00759f4ff69c20e055c8ae/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/8ffab6826544c208ef2427a17cf417af34b43d2b54fa3898bc9ba80422aa5bfc/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/d86e4a7b645c2db613594e12de8f72710ddbff153fe15f7f9039d69e79bf9031/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/a786c82618cfa0590b7abb08ba5e2d6ec70636d1c56b274256f702968602d21c/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/3e9c677b86f72cc7477fa0575aa1020ded2603a6b2a8d28607bc83891feaf0d6/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/0084a3df8d9bab707e5715619f3dc99854defd1be7edadecbddde40b2bbd926c/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/10927eec46d69258e1716809c501f03f67a25d9d99e68651334367edaabfa565/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/1f8b57b37be1cd3822149dd398685626b5b5a254f9ee1a5b692e02c7ab07b0a8/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/08369f30357638b6a42a87c86571ed5acb88e97ceaeda330a3c8aa2a05de8e47/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/381ac58a63bb4bc1e3ae5b154be1f8f33df76977755551d87f0e1d97f8e321e4/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/72c625829c1280ffb98f6c413dbcf70303372cc12762a9ce11ce86be6ea306b2/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/41dedfbcdbbe425cf3cf1038f03a29ec7c188d74f0d0455c1136087b274fbb71/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/4bc106ecdce181d2c3d9934ba0f04450c0a0306441bbfdca9a6d6462a9ca2da3/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/1f8d9172bf5505ca1a5cc6a8dbc162bee2d49313d7894f2ef036b6bba0d69625/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/9c60d483e304e77c201355f20e2de531361ddc54326f426653ea197a946572f2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/589ab5b31c0714ffde38f80a2f2d5f52fb69248622ea6c233f5f751aeafb7bac/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/65a420b8f4367d04411bedfb5a60647d576cbb45a116f520e2c5b47c2b16f482/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/16d504d48408c0da91b1c33364d346e3cb6b6b5c4c5ec177edbc04b4a5d0d91b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2f0ad4d4e6b50b40afe287b24894efd4bbb16ad64c89d23290bf3a5734cdf701/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f3453df0134de7fd828deb389de24ad47d267d7e0f03ac7a246d81d7968de35b/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 12:28:09.243645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.243214 6727 manager.go:217] Machine: {Timestamp:2026-10-31 12:28:09.242763484 +0000 UTC m=+0.057777865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098903 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c75ab32dc0212b543d01a00ad297 SystemUUID:ec21c75a-b32d-c021-2b54-3d01a00ad297 BootID:53aae745-adf1-4ced-92d1-b9b46ae701f1 Filesystems:[{Device:/var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b626d907-7b4b-4437-9e69-a82333ebf268/volumes/kubernetes.io~projected/kube-api-access-j6w2l DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c0e28ad18a3cf4138d38ee49042564bc6d06463c1742785f511280265b3cc14/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c41fab2e-6e79-4fe1-a46e-996b4012efdf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/052e1a61-3220-4a91-83c5-a08bb829f358/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/052e1a61-3220-4a91-83c5-a08bb829f358/volumes/kubernetes.io~projected/kube-api-access-qdpk5 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~projected/kube-api-access-2t7qm DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53f674e2f0d19c82af83f0eb57c86faaefdf5ded97fb11ee9e0cd179cb56ebaf/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a428dff-76d1-44aa-864a-2532953ad0ad/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:469 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c41fab2e-6e79-4fe1-a46e-996b4012efdf/volumes/kubernetes.io~projected/kube-api-access-wk9c6 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aca7a2bbb04671497f30cbb3751e89c0db6e7b379e60eb22f1854b16a8dacac0/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c742a42beed1730c1859e9d6b22dadf86deb83a71a721a68b5cba1916362961f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ee0417534535f6be00ca2415e5591f4371166f1e1fdca80eb59b3157986c752/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a95a2e68-c596-4c2e-b1bd-20d31bb31919/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~projected/kube-api-access-z4f5z DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:362 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~projected/kube-api-access-2vx5z DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c899e55c08191e87dc2980e9d90e93ed51866455f80ef6c79196ad68cfa65f9/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a08818c28770ab580328f490ea04deab9b6644b8995a84faedf6e482d70b3bd5/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a632a023-665e-4567-b2fc-13b7c37d152d/volumes/kubernetes.io~projected/kube-api-access-pw92g DeviceMajor:0 DeviceMinor:474 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b1925581-bb98-4bd2-8c66-5cafa370dfc9/volumes/kubernetes.io~projected/kube-api-access-r2dbt DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/6d278316-af3e-4b36-8505-a5d464016e85/volumes/kubernetes.io~projected/kube-api-access-zvgcz DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d97b652f-9d17-4078-90ea-ac9bd9820384/volumes/kubernetes.io~projected/kube-api-access-ppm8b DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c68c679ea306addeee1bd266a45a59452fb0ac79e7d0cb157e93aa62a29e0048/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2726bde3-96ae-4717-8732-f7e792a0ddfb/volumes/kubernetes.io~projected/kube-api-access-p7tf9 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3b15170a311240f888e8ec6647dec45c00df493eababc9b7b485e847c28cf22/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:262 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2726bde3-96ae-4717-8732-f7e792a0ddfb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a428dff-76d1-44aa-864a-2532953ad0ad/volumes/kubernetes.io~projected/kube-api-access-djjrr DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad2b3ee2-f439-49ca-a106-63e231dc232f/volumes/kubernetes.io~projected/kube-api-access-7qglj DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82c724bb90c5850cdda5082faa02beba25df46a91866906e87b98d72f7f0fb83/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38261aa230e7c9dfecbef1ad3e00e6747ff2cbfb43dafec7c7da608c7151da95/userdata/shm DeviceMajor:0 DeviceMinor:479 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:365 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:267 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~projected/kube-api-access-79d28 DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32afbb71-9024-4d12-a27a-994892cf38e4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~projected/kube-api-access-qcsz9 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82ecc097-1c88-4f8c-928c-708b9a628a72/volumes/kubernetes.io~projected/kube-api-access-z79d2 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:267 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05afc9f1f06d7ed71042fbdbcc30ea7c79e1e53d369ff360e68405238fd537d3/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d06e6e9-51c0-403a-8ea0-ecc9de005803/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:368 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70a4a90f0770abbc2d635d6b7a7987756c9c0d9357e3eec443ef0a7ea96a15a9/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:473 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:261 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e15db8cdab7ba7373a055383a82face00539d0e5cc00432bad7c3c623fcf5ad8/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:292 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4973ac2f763667fa8ca041361dc1eb087b5298b42d35c0496c943af4ea0629b/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~projected/kube-api-access-jbr8s DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3522e1a2df5c37983867d0146f13001a3dcbf177d6d255db4f1c587270bb701/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d06e6e9-51c0-403a-8ea0-ecc9de005803/volumes/kubernetes.io~projected/kube-api-access-p7kdf DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9a67f7b03a671338555684404a3efa934cefa0e6e14f71c12515c7b624bfded/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59359b45872d4afd13eeec5f538782c767ad10174592d8bccaffe1799987ed97/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:367 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/958a8aee-281c-45d9-8c87-ee2ce835fbee/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d436ac1dd286e85aa10a55cdbd5f2b547f728710da521dcf6b7476fb0db35d8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e7f16d-ec14-40bd-908f-272016e1464f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes/kubernetes.io~projected/kube-api-access-ckfkv DeviceMajor:0 DeviceMinor:263 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fc0c5ec-9758-40ff-9990-ab568b8d02dd/volumes/kubernetes.io~projected/kube-api-access-8vzgv DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3950d83734febb3fbfa39c130771e6be499b0cfe63820b169f74db1d5e37c98/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0a973f-9d43-4bec-a134-eda7d2b7a052/volumes/kubernetes.io~projected/kube-api-access-wxnw6 DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c6a8c24ebc68aa823920ce33f478657d2b6b193a264d2606abe6198d8fe66db/userdata/shm DeviceMajor:0 DeviceMinor:475 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:05afc9f1f06d7ed MacAddress:b6:db:81:6f:a1:26 Speed:10000 Mtu:8901} {Name:38261aa230e7c9d MacAddress:6e:4c:1b:4d:fb:1d Speed:10000 Mtu:8901} {Name:53f674e2f0d19c8 MacAddress:02:42:73:c2:a1:6e Speed:10000 Mtu:8901} {Name:59359b45872d4af MacAddress:26:5e:0f:49:71:cc Speed:10000 Mtu:8901} {Name:5cd6628df84188e MacAddress:e6:b4:64:62:79:c3 Speed:10000 Mtu:8901} {Name:6c6a8c24ebc68aa MacAddress:fa:b3:53:1c:f0:cd Speed:10000 Mtu:8901} {Name:6ee0417534535f6 MacAddress:4e:d5:15:bb:cf:f8 Speed:10000 Mtu:8901} {Name:70a4a90f0770abb MacAddress:5a:aa:c1:07:84:69 Speed:10000 Mtu:8901} {Name:82c724bb90c5850 MacAddress:e2:46:62:22:b5:c8 Speed:10000 Mtu:8901} {Name:a08818c28770ab5 MacAddress:f6:2e:c8:d1:0a:31 Speed:10000 Mtu:8901} {Name:b4973ac2f763667 MacAddress:f2:29:35:3f:e7:26 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e8:a3:4f:9b Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:ef:26:07:c0:fa Speed:0 Mtu:8901} {Name:d3b15170a311240 MacAddress:fe:cb:4e:7a:23:ef Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e8:a3:4f:9b Speed:0 Mtu:9001} {Name:f9a67f7b03a6713 MacAddress:ba:5e:b5:26:69:99 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b2:a1:29:42:59:6f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:5d:4b:f1:c0:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:28:09.243894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.243651 6727 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:28:09.243894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.243694 6727 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:28:09.243971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.243949 6727 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:28:09.244085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.243972 6727 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-45.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:28:09.244128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244087 6727 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:28:09.244128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244094 6727 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:28:09.244128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244105 6727 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:28:09.244128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244117 6727 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:28:09.244228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244188 6727 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:28:09.244337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244328 6727 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:28:09.244429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244421 6727 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:28:09.244474 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244456 6727 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:28:09.244503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244488 6727 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:28:09.244531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244503 6727 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:28:09.244531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244515 6727 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:28:09.244531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.244525 6727 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:28:09.245633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.245613 6727 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:28:09.245686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.245634 6727 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:28:09.246137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.246118 6727 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:28:09.249807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.249794 6727 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:28:09.249807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.249807 6727 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:28:09.249907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.249898 6727 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 12:28:09.250099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250086 6727 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:28:09.250157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250115 6727 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:28:09.250222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250212 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250226 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250232 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250239 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250244 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250250 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250259 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250267 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:28:09.250271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250273 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:28:09.250666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250278 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:28:09.250666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250298 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:28:09.250666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250308 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:28:09.250666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250332 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:28:09.250666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.250340 6727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:28:09.253674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.253662 6727 server.go:1263] "Started kubelet" Oct 31 12:28:09.254477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.253736 6727 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:28:09.254477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.253777 6727 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:28:09.254477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.253853 6727 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:28:09.254477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.254277 6727 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:28:09.254221 ip-10-0-142-45 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:28:09.255514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.255500 6727 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:28:09.259199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.259180 6727 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.259205 6727 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.259295 6727 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.259307 6727 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:09.259367 6727 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-45.us-east-2.compute.internal\" not found" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.259392 6727 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.259377 6727 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:59 +0000 UTC" deadline="2028-04-14 17:41:44.359792215 +0000 UTC" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.259407 6727 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12749h13m35.100388207s" Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.260496 6727 factory.go:55] Registering systemd factory Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.260512 6727 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.260714 6727 factory.go:153] Registering CRI-O factory Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.260725 6727 factory.go:223] Registration of the crio container factory successfully Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.260781 6727 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.260806 6727 factory.go:103] Registering Raw factory Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.260821 6727 manager.go:1199] Started watching for new ooms in manager Oct 31 12:28:09.261334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.261113 6727 manager.go:319] Starting recovery of all containers Oct 31 12:28:09.262210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262175 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" volumeName="kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262213 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" volumeName="kubernetes.io/projected/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-kube-api-access-z4f5z" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262226 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d97b652f-9d17-4078-90ea-ac9bd9820384" volumeName="kubernetes.io/secret/d97b652f-9d17-4078-90ea-ac9bd9820384-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262239 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262250 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262261 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262273 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2726bde3-96ae-4717-8732-f7e792a0ddfb" volumeName="kubernetes.io/secret/2726bde3-96ae-4717-8732-f7e792a0ddfb-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262285 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262300 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262311 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a95a2e68-c596-4c2e-b1bd-20d31bb31919" volumeName="kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262322 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad2b3ee2-f439-49ca-a106-63e231dc232f" volumeName="kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262334 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262346 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262366 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262380 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32afbb71-9024-4d12-a27a-994892cf38e4" volumeName="kubernetes.io/secret/32afbb71-9024-4d12-a27a-994892cf38e4-agent-certs" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262392 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262402 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0a973f-9d43-4bec-a134-eda7d2b7a052" volumeName="kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262413 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="958a8aee-281c-45d9-8c87-ee2ce835fbee" volumeName="kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-tmp" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262423 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b626d907-7b4b-4437-9e69-a82333ebf268" volumeName="kubernetes.io/projected/b626d907-7b4b-4437-9e69-a82333ebf268-kube-api-access-j6w2l" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262439 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262450 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262518 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262530 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fc0c5ec-9758-40ff-9990-ab568b8d02dd" volumeName="kubernetes.io/configmap/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-serviceca" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262541 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fc0c5ec-9758-40ff-9990-ab568b8d02dd" volumeName="kubernetes.io/projected/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-kube-api-access-8vzgv" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262551 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" volumeName="kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262576 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262587 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" volumeName="kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262599 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262614 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262638 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a166dd5f-2e0c-45a0-8dc6-692741a8e1b3" volumeName="kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262650 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b626d907-7b4b-4437-9e69-a82333ebf268" volumeName="kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-multus-daemon-config" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262662 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b626d907-7b4b-4437-9e69-a82333ebf268" volumeName="kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-cni-binary-copy" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262675 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" volumeName="kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262686 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/configmap/d2e7f16d-ec14-40bd-908f-272016e1464f-metrics-client-ca" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262697 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" volumeName="kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262709 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262720 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad2b3ee2-f439-49ca-a106-63e231dc232f" volumeName="kubernetes.io/projected/ad2b3ee2-f439-49ca-a106-63e231dc232f-kube-api-access-7qglj" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262732 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" volumeName="kubernetes.io/empty-dir/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-textfile" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262743 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d97b652f-9d17-4078-90ea-ac9bd9820384" volumeName="kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-script-lib" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262753 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d97b652f-9d17-4078-90ea-ac9bd9820384" volumeName="kubernetes.io/projected/d97b652f-9d17-4078-90ea-ac9bd9820384-kube-api-access-ppm8b" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262765 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262775 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262787 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d278316-af3e-4b36-8505-a5d464016e85" volumeName="kubernetes.io/projected/6d278316-af3e-4b36-8505-a5d464016e85-kube-api-access-zvgcz" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262800 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0a973f-9d43-4bec-a134-eda7d2b7a052" volumeName="kubernetes.io/projected/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-api-access-wxnw6" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262810 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ecc097-1c88-4f8c-928c-708b9a628a72" volumeName="kubernetes.io/projected/82ecc097-1c88-4f8c-928c-708b9a628a72-kube-api-access-z79d2" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262823 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad2b3ee2-f439-49ca-a106-63e231dc232f" volumeName="kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-binary-copy" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262835 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" volumeName="kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262847 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262857 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="052e1a61-3220-4a91-83c5-a08bb829f358" volumeName="kubernetes.io/configmap/052e1a61-3220-4a91-83c5-a08bb829f358-config-volume" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262869 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="052e1a61-3220-4a91-83c5-a08bb829f358" volumeName="kubernetes.io/projected/052e1a61-3220-4a91-83c5-a08bb829f358-kube-api-access-qdpk5" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262879 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262891 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2726bde3-96ae-4717-8732-f7e792a0ddfb" volumeName="kubernetes.io/configmap/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262903 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262915 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262926 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262936 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262957 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262968 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262980 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.262992 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c41fab2e-6e79-4fe1-a46e-996b4012efdf" volumeName="kubernetes.io/secret/c41fab2e-6e79-4fe1-a46e-996b4012efdf-cert" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263003 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/projected/d2e7f16d-ec14-40bd-908f-272016e1464f-kube-api-access-jbr8s" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263016 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" volumeName="kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:28:09.262987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263027 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d97b652f-9d17-4078-90ea-ac9bd9820384" volumeName="kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-config" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263038 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263059 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263073 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263083 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a428dff-76d1-44aa-864a-2532953ad0ad" volumeName="kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263095 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="958a8aee-281c-45d9-8c87-ee2ce835fbee" volumeName="kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-tuned" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263114 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad2b3ee2-f439-49ca-a106-63e231dc232f" volumeName="kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263126 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263137 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" volumeName="kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-metrics-client-ca" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263147 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d06e6e9-51c0-403a-8ea0-ecc9de005803" volumeName="kubernetes.io/projected/0d06e6e9-51c0-403a-8ea0-ecc9de005803-kube-api-access-p7kdf" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263159 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263171 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" volumeName="kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263181 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c41fab2e-6e79-4fe1-a46e-996b4012efdf" volumeName="kubernetes.io/projected/c41fab2e-6e79-4fe1-a46e-996b4012efdf-kube-api-access-wk9c6" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263192 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d278316-af3e-4b36-8505-a5d464016e85" volumeName="kubernetes.io/empty-dir/6d278316-af3e-4b36-8505-a5d464016e85-tmp-dir" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263202 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0a973f-9d43-4bec-a134-eda7d2b7a052" volumeName="kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263215 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/empty-dir/d2e7f16d-ec14-40bd-908f-272016e1464f-tmp" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263225 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="052e1a61-3220-4a91-83c5-a08bb829f358" volumeName="kubernetes.io/secret/052e1a61-3220-4a91-83c5-a08bb829f358-metrics-tls" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263237 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263248 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="958a8aee-281c-45d9-8c87-ee2ce835fbee" volumeName="kubernetes.io/projected/958a8aee-281c-45d9-8c87-ee2ce835fbee-kube-api-access-qcsz9" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263259 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1925581-bb98-4bd2-8c66-5cafa370dfc9" volumeName="kubernetes.io/projected/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kube-api-access-r2dbt" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263269 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263281 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d97b652f-9d17-4078-90ea-ac9bd9820384" volumeName="kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-env-overrides" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263296 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="052e1a61-3220-4a91-83c5-a08bb829f358" volumeName="kubernetes.io/empty-dir/052e1a61-3220-4a91-83c5-a08bb829f358-tmp-dir" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263308 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263319 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a95a2e68-c596-4c2e-b1bd-20d31bb31919" volumeName="kubernetes.io/configmap/a95a2e68-c596-4c2e-b1bd-20d31bb31919-metrics-client-ca" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263329 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a95a2e68-c596-4c2e-b1bd-20d31bb31919" volumeName="kubernetes.io/projected/a95a2e68-c596-4c2e-b1bd-20d31bb31919-kube-api-access-2t7qm" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263340 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" volumeName="kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263351 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263362 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32afbb71-9024-4d12-a27a-994892cf38e4" volumeName="kubernetes.io/configmap/32afbb71-9024-4d12-a27a-994892cf38e4-konnectivity-ca" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263375 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a428dff-76d1-44aa-864a-2532953ad0ad" volumeName="kubernetes.io/projected/7a428dff-76d1-44aa-864a-2532953ad0ad-kube-api-access-djjrr" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263386 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ecc097-1c88-4f8c-928c-708b9a628a72" volumeName="kubernetes.io/configmap/82ecc097-1c88-4f8c-928c-708b9a628a72-iptables-alerter-script" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263399 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a95a2e68-c596-4c2e-b1bd-20d31bb31919" volumeName="kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263410 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" volumeName="kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263422 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e7f16d-ec14-40bd-908f-272016e1464f" volumeName="kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-grpc-tls" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263434 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d06e6e9-51c0-403a-8ea0-ecc9de005803" volumeName="kubernetes.io/secret/0d06e6e9-51c0-403a-8ea0-ecc9de005803-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263448 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2726bde3-96ae-4717-8732-f7e792a0ddfb" volumeName="kubernetes.io/projected/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-api-access-p7tf9" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263475 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac5a328-04f5-4857-bcc8-e081c18907d3" volumeName="kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263488 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0a973f-9d43-4bec-a134-eda7d2b7a052" volumeName="kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-metrics-client-ca" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263499 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0a973f-9d43-4bec-a134-eda7d2b7a052" volumeName="kubernetes.io/empty-dir/7c0a973f-9d43-4bec-a134-eda7d2b7a052-volume-directive-shadow" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263511 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d26075f-7f96-4d14-bf9e-85ec07189e38" volumeName="kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263521 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2726bde3-96ae-4717-8732-f7e792a0ddfb" volumeName="kubernetes.io/empty-dir/2726bde3-96ae-4717-8732-f7e792a0ddfb-data-volume" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263533 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0a973f-9d43-4bec-a134-eda7d2b7a052" volumeName="kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263543 6727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a632a023-665e-4567-b2fc-13b7c37d152d" volumeName="kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g" seLinuxMountContext="" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263556 6727 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:28:09.265994 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.263564 6727 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:28:09.267884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.266442 6727 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:28:09.267884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.266765 6727 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:28:09.269233 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.269097 6727 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 12:28:09.269723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.269706 6727 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:28:09.290731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.290701 6727 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:28:09.292346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.292329 6727 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:28:09.292346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.292346 6727 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:28:09.292504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.292365 6727 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:28:09.292504 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:09.292401 6727 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 12:28:09.296036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.296016 6727 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:28:09.311777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.311754 6727 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" exitCode=0 Oct 31 12:28:09.320421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.320397 6727 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="83cb5584f2eb69184127e902ce3c77c7e77dba7a973cfc694989f669b8245ca7" exitCode=0 Oct 31 12:28:09.320421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.320416 6727 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="1d48bc66fae186e82afa50e31a748128531045c389d151baf59431facc71c819" exitCode=0 Oct 31 12:28:09.320421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.320425 6727 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="3c58748ca8f48a956507ee3820dcb63edb2d8bdd6cac16af9f68507f507dbd36" exitCode=0 Oct 31 12:28:09.320602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.320433 6727 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="dd9bda1f4a04a917f1eb060a7b30f5a112d52ccfc5b8757e8de71960f1d169ea" exitCode=0 Oct 31 12:28:09.320602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.320441 6727 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="a50140f0da59749fe2a68810e7757f6feaf607437e2228bddf75d1a793e1265c" exitCode=0 Oct 31 12:28:09.320602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.320447 6727 generic.go:356] "Generic (PLEG): container finished" podID="ad2b3ee2-f439-49ca-a106-63e231dc232f" containerID="a8ad48553cb66f49747c49a7d5a85c4f3a8ad146659f322ed7f8ce72cfa52284" exitCode=0 Oct 31 12:28:09.329376 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.329334 6727 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937" exitCode=0 Oct 31 12:28:09.351152 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.351129 6727 generic.go:356] "Generic (PLEG): container finished" podID="d86c20a2519affed9d4fc6daf0172b42" containerID="7e758f589390f8bcae257e4c24d0d7f8679b06ecdcf5eedfd7a9d06bb9ad441c" exitCode=0 Oct 31 12:28:09.354943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.354920 6727 generic.go:356] "Generic (PLEG): container finished" podID="d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105" containerID="6b77e7c2782f2ae913e1ba52e814ce6096cbcc9afda05fe0966f058465746cf2" exitCode=0 Oct 31 12:28:09.374550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.374537 6727 manager.go:324] Recovery completed Oct 31 12:28:09.392505 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:09.392445 6727 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 12:28:09.398336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398319 6727 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:28:09.398336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398333 6727 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:28:09.398436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398348 6727 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:28:09.398485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398447 6727 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 12:28:09.398874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398458 6727 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 12:28:09.398913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398875 6727 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 12:28:09.398913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398882 6727 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 12:28:09.398913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398888 6727 policy_none.go:50] "Start" Oct 31 12:28:09.398913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398897 6727 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:28:09.398913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.398908 6727 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:28:09.399059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.399032 6727 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 12:28:09.399059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.399039 6727 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 12:28:09.399059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.399044 6727 policy_none.go:44] "Start" Oct 31 12:28:09.403802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.403773 6727 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:28:09.406399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.406388 6727 manager.go:341] "Starting Device Plugin manager" Oct 31 12:28:09.406434 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:09.406410 6727 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:28:09.406434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.406418 6727 server.go:87] "Starting device plugin registration server" Oct 31 12:28:09.406880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.406748 6727 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:28:09.407010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.406982 6727 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:28:09.407259 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.407240 6727 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:28:09.407384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.407351 6727 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:28:09.407436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.407394 6727 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:28:09.407436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.407401 6727 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:28:09.407590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.407474 6727 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:28:09.407382717Z","UUID":"e66e262d-6e7e-44c6-96a5-be25944dc17f","Handler":null,"Name":"","Endpoint":""} Oct 31 12:28:09.411142 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:09.411120 6727 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:28:09.411979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.411960 6727 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:28:09.412059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.411998 6727 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:28:09.508116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.508092 6727 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:28:09.509970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.509954 6727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:28:09.510042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.509982 6727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:28:09.510042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.509992 6727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-45.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:28:09.510042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.510038 6727 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.521643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.521623 6727 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.521705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.521673 6727 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.592796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.592751 6727 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal"] Oct 31 12:28:09.593400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.593386 6727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.593504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.593449 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"2e9561195d04d47702d33d861addb8e7","Type":"ContainerStarted","Data":"61996ca91cb34a11b03f37f9d73e12798a624d90de32e9b5ebe4131ac3a3b602"} Oct 31 12:28:09.593547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.593514 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"2e9561195d04d47702d33d861addb8e7","Type":"ContainerStarted","Data":"bf0343f74383d952607cc0125e819481ecfad2c6884e4831f54ae99566d8b091"} Oct 31 12:28:09.593547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.593528 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"d86c20a2519affed9d4fc6daf0172b42","Type":"ContainerStarted","Data":"2f9edf796eb9ca6a7d35f0a4b3fda0d378fdd1875cf11d84c9b5787daceacabd"} Oct 31 12:28:09.593547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.593545 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"d86c20a2519affed9d4fc6daf0172b42","Type":"ContainerDied","Data":"7e758f589390f8bcae257e4c24d0d7f8679b06ecdcf5eedfd7a9d06bb9ad441c"} Oct 31 12:28:09.593647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.593558 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" event={"ID":"d86c20a2519affed9d4fc6daf0172b42","Type":"ContainerStarted","Data":"c5e449154fda136a604926c02a8b1ef932770d5b83589249bc344cd4f34ed073"} Oct 31 12:28:09.593647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.593635 6727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.605318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.605298 6727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:28:09.605424 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:09.605362 6727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.605759 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.605744 6727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:28:09.605809 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:09.605792 6727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.671063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.671012 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9561195d04d47702d33d861addb8e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"2e9561195d04d47702d33d861addb8e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.671135 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.671069 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.671135 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.671086 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.772050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.772022 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.772050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.772045 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.772193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.772059 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.772193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.772078 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9561195d04d47702d33d861addb8e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"2e9561195d04d47702d33d861addb8e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.772193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.772113 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9561195d04d47702d33d861addb8e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"2e9561195d04d47702d33d861addb8e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:09.772193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:09.772122 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d86c20a2519affed9d4fc6daf0172b42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" (UID: \"d86c20a2519affed9d4fc6daf0172b42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:10.244846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.244821 6727 apiserver.go:52] "Watching apiserver" Oct 31 12:28:10.253431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.253409 6727 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:28:10.254134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.254106 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57dbb8df47-wq5vh","openshift-dns/dns-default-4jx6k","openshift-ingress-canary/ingress-canary-jlbzw","openshift-multus/network-metrics-daemon-kr4dj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh","openshift-insights/insights-runtime-extractor-b6x24","openshift-monitoring/thanos-querier-577cf77cd5-p495x","kube-system/konnectivity-agent-6b687","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/node-exporter-67z7l","openshift-network-operator/iptables-alerter-s2wpv","openshift-ovn-kubernetes/ovnkube-node-jq9k9","openshift-dns/node-resolver-vjlsm","openshift-image-registry/node-ca-dz7qv","kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-g7zbh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-wnnvc","openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl","openshift-monitoring/monitoring-plugin-77f4f76594-wqp22","openshift-network-diagnostics/network-check-target-zcknp","openshift-multus/multus-p2ppd","openshift-cluster-node-tuning-operator/tuned-sv5jl","openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw"] Oct 31 12:28:10.259916 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.259891 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:28:10.260021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.259953 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ljk6\"" Oct 31 12:28:10.260021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260007 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bbb6f\"" Oct 31 12:28:10.260126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260032 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\"" Oct 31 12:28:10.260126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260007 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kldqp\"" Oct 31 12:28:10.260126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260092 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\"" Oct 31 12:28:10.260282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260181 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:28:10.260282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260195 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x5rvd\"" Oct 31 12:28:10.260823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260792 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-df646\"" Oct 31 12:28:10.260934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.260908 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:28:10.261070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.261047 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m5kv7\"" Oct 31 12:28:10.261546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.261520 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\"" Oct 31 12:28:10.261912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.261891 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:28:10.261992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.261965 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9d2j7\"" Oct 31 12:28:10.262563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.262540 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:28:10.262846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.262825 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.262918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.262899 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.263050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263030 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.263151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263134 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.263204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263190 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ng46x\"" Oct 31 12:28:10.263252 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263243 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.263301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263288 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:28:10.263420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263402 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:28:10.263504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263493 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:28:10.263714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263695 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.263836 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263813 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.263959 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263940 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.264041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264023 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:28:10.264094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264046 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.264162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263494 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.264222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264169 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.264222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264012 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:28:10.264317 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264296 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.264366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264350 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.264419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.263951 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:28:10.264482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264424 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzb2l\"" Oct 31 12:28:10.264756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264734 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.264834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264819 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c6kzr\"" Oct 31 12:28:10.264975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264956 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:28:10.264975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.264972 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.265076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265002 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:28:10.265143 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265126 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:28:10.265193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265146 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.265248 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265236 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:28:10.265380 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265361 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.265437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265382 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.265437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265425 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:28:10.265554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265529 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nzzrg\"" Oct 31 12:28:10.266016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265240 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:28:10.266016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265652 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:28:10.266016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265739 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.266016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.265822 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.271815 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.271795 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 12:28:10.271913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.271801 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9c5q9\"" Oct 31 12:28:10.271913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.271845 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:28:10.271913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.271845 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcqch\"" Oct 31 12:28:10.271913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.271863 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 12:28:10.271913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.271893 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:28:10.272131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.271964 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:28:10.273922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.273900 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.273961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.273942 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-log-socket\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.273999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.273968 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9c6\" (UniqueName: \"kubernetes.io/projected/c41fab2e-6e79-4fe1-a46e-996b4012efdf-kube-api-access-wk9c6\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:28:10.274138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274121 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-kubelet-config\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.274174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274150 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-sys\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.274174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274168 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-sys-fs\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.274249 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274184 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.274249 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274235 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.274326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274253 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-host\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:28:10.274326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274292 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzgv\" (UniqueName: \"kubernetes.io/projected/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-kube-api-access-8vzgv\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:28:10.274326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274321 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d278316-af3e-4b36-8505-a5d464016e85-hosts-file\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:28:10.274425 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274351 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-conf\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.274480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274426 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-run\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.274514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274481 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.274554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274522 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dbt\" (UniqueName: \"kubernetes.io/projected/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kube-api-access-r2dbt\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.274587 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274552 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-kubelet\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.274751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274732 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-etc-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.274802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274770 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-bin\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.274836 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274820 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-env-overrides\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.274876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274863 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-host\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.274924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274896 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7tf9\" (UniqueName: \"kubernetes.io/projected/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-api-access-p7tf9\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.274980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.274969 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-cnibin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.275083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275056 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-env-overrides\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.275139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275067 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-bin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.275139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275111 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-multus\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.275229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275137 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-etc-kubernetes\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.275229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275164 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-cnibin\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.275296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275228 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.275296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275273 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-slash\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.275361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275298 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-ovn\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.275361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275319 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2726bde3-96ae-4717-8732-f7e792a0ddfb-data-volume\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.275361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275343 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ecc097-1c88-4f8c-928c-708b9a628a72-host-slash\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:28:10.275489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275366 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-multus-daemon-config\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.275489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275404 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.275489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275428 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-multus-certs\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.275489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275423 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2726bde3-96ae-4717-8732-f7e792a0ddfb-data-volume\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.275603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275497 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.275603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275589 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-multus-daemon-config\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.275674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275660 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qglj\" (UniqueName: \"kubernetes.io/projected/ad2b3ee2-f439-49ca-a106-63e231dc232f-kube-api-access-7qglj\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.275704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275696 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-var-lib-kubelet\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.275908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275720 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-var-lib-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.275908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275745 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.275908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275788 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-netd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.275908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275817 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.275908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275838 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79d2\" (UniqueName: \"kubernetes.io/projected/82ecc097-1c88-4f8c-928c-708b9a628a72-kube-api-access-z79d2\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:28:10.275908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275855 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-original-pull-secret\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.275908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275872 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c41fab2e-6e79-4fe1-a46e-996b4012efdf-cert\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275924 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-hostroot\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.275978 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276023 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82ecc097-1c88-4f8c-928c-708b9a628a72-iptables-alerter-script\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276040 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysconfig\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276092 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsz9\" (UniqueName: \"kubernetes.io/projected/958a8aee-281c-45d9-8c87-ee2ce835fbee-kube-api-access-qcsz9\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276136 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276161 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c41fab2e-6e79-4fe1-a46e-996b4012efdf-cert\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276162 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-netns\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.276231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276202 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6w2l\" (UniqueName: \"kubernetes.io/projected/b626d907-7b4b-4437-9e69-a82333ebf268-kube-api-access-j6w2l\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276250 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82ecc097-1c88-4f8c-928c-708b9a628a72-iptables-alerter-script\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276358 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-dbus\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276393 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276433 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-lib-modules\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276477 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjrr\" (UniqueName: \"kubernetes.io/projected/7a428dff-76d1-44aa-864a-2532953ad0ad-kube-api-access-djjrr\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276506 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-node-log\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276542 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97b652f-9d17-4078-90ea-ac9bd9820384-ovn-node-metrics-cert\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276583 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-script-lib\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276607 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2b3ee2-f439-49ca-a106-63e231dc232f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.276649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276631 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276658 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-registration-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276686 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-conf-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276710 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276718 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97b652f-9d17-4078-90ea-ac9bd9820384-ovn-node-metrics-cert\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276736 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-tuned\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276779 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppm8b\" (UniqueName: \"kubernetes.io/projected/d97b652f-9d17-4078-90ea-ac9bd9820384-kube-api-access-ppm8b\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276805 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-script-lib\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276817 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-tuned\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276845 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgcz\" (UniqueName: \"kubernetes.io/projected/6d278316-af3e-4b36-8505-a5d464016e85-kube-api-access-zvgcz\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276865 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276880 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-device-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276896 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276938 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-config\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.276998 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-system-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277032 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-socket-dir-parent\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277059 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-kubelet\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277092 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-modprobe-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277129 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277147 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-socket-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277163 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-systemd-units\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277183 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97b652f-9d17-4078-90ea-ac9bd9820384-ovnkube-config\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277185 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-systemd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277222 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-cni-binary-copy\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277247 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-systemd\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277320 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32afbb71-9024-4d12-a27a-994892cf38e4-konnectivity-ca\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277330 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a428dff-76d1-44aa-864a-2532953ad0ad-metrics-certs\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277355 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-kubernetes\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277379 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277390 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b626d907-7b4b-4437-9e69-a82333ebf268-cni-binary-copy\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277407 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2726bde3-96ae-4717-8732-f7e792a0ddfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277458 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-serviceca\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277508 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32afbb71-9024-4d12-a27a-994892cf38e4-konnectivity-ca\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277569 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-k8s-cni-cncf-io\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277633 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2726bde3-96ae-4717-8732-f7e792a0ddfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.277668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277647 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-netns\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277685 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-system-cni-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277708 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-tmp\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277726 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32afbb71-9024-4d12-a27a-994892cf38e4-agent-certs\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277769 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2726bde3-96ae-4717-8732-f7e792a0ddfb-crio-socket\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277826 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958a8aee-281c-45d9-8c87-ee2ce835fbee-tmp\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277826 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d278316-af3e-4b36-8505-a5d464016e85-tmp-dir\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277901 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d278316-af3e-4b36-8505-a5d464016e85-tmp-dir\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.277947 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32afbb71-9024-4d12-a27a-994892cf38e4-agent-certs\") pod \"konnectivity-agent-6b687\" (UID: \"32afbb71-9024-4d12-a27a-994892cf38e4\") " pod="kube-system/konnectivity-agent-6b687" Oct 31 12:28:10.278191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.278150 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-serviceca\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:28:10.280117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.280103 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:28:10.281151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.281136 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:28:10.283218 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.283205 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 12:28:10.287907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.287888 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 12:28:10.299915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.299892 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 12:28:10.301418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.301402 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 12:28:10.306275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.306258 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79d2\" (UniqueName: \"kubernetes.io/projected/82ecc097-1c88-4f8c-928c-708b9a628a72-kube-api-access-z79d2\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:28:10.306353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.306255 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppm8b\" (UniqueName: \"kubernetes.io/projected/d97b652f-9d17-4078-90ea-ac9bd9820384-kube-api-access-ppm8b\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.307422 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.307330 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6w2l\" (UniqueName: \"kubernetes.io/projected/b626d907-7b4b-4437-9e69-a82333ebf268-kube-api-access-j6w2l\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.307422 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.307403 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qglj\" (UniqueName: \"kubernetes.io/projected/ad2b3ee2-f439-49ca-a106-63e231dc232f-kube-api-access-7qglj\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.307581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.307511 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dbt\" (UniqueName: \"kubernetes.io/projected/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kube-api-access-r2dbt\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.308214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.308193 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjrr\" (UniqueName: \"kubernetes.io/projected/7a428dff-76d1-44aa-864a-2532953ad0ad-kube-api-access-djjrr\") pod \"network-metrics-daemon-kr4dj\" (UID: \"7a428dff-76d1-44aa-864a-2532953ad0ad\") " pod="openshift-multus/network-metrics-daemon-kr4dj" Oct 31 12:28:10.308300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.308283 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw92g\" (UniqueName: \"kubernetes.io/projected/a632a023-665e-4567-b2fc-13b7c37d152d-kube-api-access-pw92g\") pod \"network-check-target-zcknp\" (UID: \"a632a023-665e-4567-b2fc-13b7c37d152d\") " pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:28:10.308410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.308394 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzgv\" (UniqueName: \"kubernetes.io/projected/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-kube-api-access-8vzgv\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:28:10.311509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.311490 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsz9\" (UniqueName: \"kubernetes.io/projected/958a8aee-281c-45d9-8c87-ee2ce835fbee-kube-api-access-qcsz9\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.312378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.312363 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7tf9\" (UniqueName: \"kubernetes.io/projected/2726bde3-96ae-4717-8732-f7e792a0ddfb-kube-api-access-p7tf9\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.312908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.312895 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rs48x\"" Oct 31 12:28:10.313285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.312880 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:28:10.313517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.313503 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgcz\" (UniqueName: \"kubernetes.io/projected/6d278316-af3e-4b36-8505-a5d464016e85-kube-api-access-zvgcz\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:28:10.313937 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.313920 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9c6\" (UniqueName: \"kubernetes.io/projected/c41fab2e-6e79-4fe1-a46e-996b4012efdf-kube-api-access-wk9c6\") pod \"ingress-canary-jlbzw\" (UID: \"c41fab2e-6e79-4fe1-a46e-996b4012efdf\") " pod="openshift-ingress-canary/ingress-canary-jlbzw" Oct 31 12:28:10.359002 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.358980 6727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:10.359111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359098 6727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:10.359352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359297 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 12:28:10.359538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359518 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 12:28:10.359656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359604 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 12:28:10.359656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359634 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 12:28:10.359656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359644 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 12:28:10.359656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359654 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 12:28:10.359892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359685 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 12:28:10.359892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359690 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 12:28:10.359892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359708 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 12:28:10.359892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359696 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 12:28:10.359892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359750 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 12:28:10.359892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359851 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.359990 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mjk8n\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360060 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360071 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hntd5\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360112 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360165 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360166 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360181 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 12:28:10.360224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360208 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 12:28:10.360555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360260 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b7krqr7laupp8\"" Oct 31 12:28:10.360555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360277 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 12:28:10.360555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360293 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 12:28:10.360555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360340 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 12:28:10.360555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360361 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5kk9p\"" Oct 31 12:28:10.360555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360407 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 12:28:10.360907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.360880 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6dgmh\"" Oct 31 12:28:10.361036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.361022 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tmrcz\"" Oct 31 12:28:10.361341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.361325 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 12:28:10.361415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.361327 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 12:28:10.361677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.361660 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 12:28:10.361902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.361888 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fh91gmt3il7ar\"" Oct 31 12:28:10.362272 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.362078 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 12:28:10.362272 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.362081 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 12:28:10.362272 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.362227 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:28:10.362584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.362565 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 12:28:10.364685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.364632 6727 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:28:10.365065 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.364997 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 12:28:10.365844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.365822 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 12:28:10.370908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.370893 6727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:28:10.370984 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.370932 6727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:10.371499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.371475 6727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:28:10.371579 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.371520 6727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-45.us-east-2.compute.internal" Oct 31 12:28:10.378041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378015 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.378133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378051 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.378133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378069 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4f5z\" (UniqueName: \"kubernetes.io/projected/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-kube-api-access-z4f5z\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.378133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378084 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.378133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378108 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-kubernetes\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.378133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378128 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-k8s-cni-cncf-io\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378150 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-system-cni-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378174 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c0a973f-9d43-4bec-a134-eda7d2b7a052-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378198 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378206 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-kubernetes\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378223 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2726bde3-96ae-4717-8732-f7e792a0ddfb-crio-socket\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378266 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-kubelet-config\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378275 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2726bde3-96ae-4717-8732-f7e792a0ddfb-crio-socket\") pod \"insights-runtime-extractor-b6x24\" (UID: \"2726bde3-96ae-4717-8732-f7e792a0ddfb\") " pod="openshift-insights/insights-runtime-extractor-b6x24" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378278 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378295 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378306 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-k8s-cni-cncf-io\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378318 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378247 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-system-cni-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378352 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c0a973f-9d43-4bec-a134-eda7d2b7a052-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378355 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-kubelet-config\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378356 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.378388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378381 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-sys\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378433 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-sys-fs\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378435 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-sys\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378473 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-host\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378499 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d278316-af3e-4b36-8505-a5d464016e85-hosts-file\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378521 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-sys-fs\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378523 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378558 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-conf\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378562 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc0c5ec-9758-40ff-9990-ab568b8d02dd-host\") pod \"node-ca-dz7qv\" (UID: \"8fc0c5ec-9758-40ff-9990-ab568b8d02dd\") " pod="openshift-image-registry/node-ca-dz7qv" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378559 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378580 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378594 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d278316-af3e-4b36-8505-a5d464016e85-hosts-file\") pod \"node-resolver-vjlsm\" (UID: \"6d278316-af3e-4b36-8505-a5d464016e85\") " pod="openshift-dns/node-resolver-vjlsm" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378607 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-cnibin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378616 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.378622 6727 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378570 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378641 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-multus\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378651 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-cnibin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378667 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-cnibin\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378690 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-multus\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378692 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378668 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-conf\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.378710 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:10.878691437 +0000 UTC m=+1.693705840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378721 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-cnibin\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378738 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-etc-kubernetes\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378762 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d06e6e9-51c0-403a-8ea0-ecc9de005803-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378797 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-etc-kubernetes\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378823 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kdf\" (UniqueName: \"kubernetes.io/projected/0d06e6e9-51c0-403a-8ea0-ecc9de005803-kube-api-access-p7kdf\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378840 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-tls\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378858 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378884 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378907 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-ovn\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378932 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-root\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378931 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378955 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-grpc-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378978 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-var-lib-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379002 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379026 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-accelerators-collector-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379028 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-ovn\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.378978 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d06e6e9-51c0-403a-8ea0-ecc9de005803-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379066 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379066 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-tls\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379074 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-var-lib-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379103 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379131 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379150 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379157 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.379148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379167 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379190 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379207 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379225 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-netns\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379258 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a95a2e68-c596-4c2e-b1bd-20d31bb31919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379268 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-grpc-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379277 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379287 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-wtmp\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379296 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379315 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-netns\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379346 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379350 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379411 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79d28\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379438 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379448 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379436 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379495 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379507 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-accelerators-collector-config\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379529 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.379538 6727 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379540 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379558 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdpk5\" (UniqueName: \"kubernetes.io/projected/052e1a61-3220-4a91-83c5-a08bb829f358-kube-api-access-qdpk5\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379581 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-device-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.379597 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:10.879580891 +0000 UTC m=+1.694595283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379618 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a95a2e68-c596-4c2e-b1bd-20d31bb31919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379627 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-device-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379628 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379667 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379675 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.379673 6727 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379712 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.379738 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:10.879722322 +0000 UTC m=+1.694736706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-config" not registered Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379767 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-modprobe-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379788 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-socket-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379803 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-systemd-units\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379819 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7qm\" (UniqueName: \"kubernetes.io/projected/a95a2e68-c596-4c2e-b1bd-20d31bb31919-kube-api-access-2t7qm\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379833 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-systemd\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379849 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-metrics-client-ca\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379866 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379872 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379844 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-socket-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379887 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-modprobe-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379902 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379905 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-systemd\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379932 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-netns\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379949 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-systemd-units\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379952 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379964 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379992 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-textfile\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.379999 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-netns\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380008 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysctl-d\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380017 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbr8s\" (UniqueName: \"kubernetes.io/projected/d2e7f16d-ec14-40bd-908f-272016e1464f-kube-api-access-jbr8s\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380044 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380060 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-textfile\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380063 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380103 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-metrics-client-ca\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380106 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380142 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380149 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380182 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380207 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/052e1a61-3220-4a91-83c5-a08bb829f358-metrics-tls\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380232 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-log-socket\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380260 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380286 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380289 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380319 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380290 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380328 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380366 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-run-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380344 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-log-socket\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380384 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380375 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380424 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380439 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380453 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-run\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380489 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-kubelet\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380504 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-etc-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380517 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380517 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-bin\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380518 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a95a2e68-c596-4c2e-b1bd-20d31bb31919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380547 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-bin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380563 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-host\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380540 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-bin\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380584 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/052e1a61-3220-4a91-83c5-a08bb829f358-metrics-tls\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380600 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-host\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380600 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-cni-bin\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380628 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-run\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380632 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380649 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-kubelet\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380632 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnw6\" (UniqueName: \"kubernetes.io/projected/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-api-access-wxnw6\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380668 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380682 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-slash\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380681 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-etc-openvswitch\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380710 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ecc097-1c88-4f8c-928c-708b9a628a72-host-slash\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380738 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-multus-certs\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380764 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380788 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380814 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380843 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-var-lib-kubelet\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380846 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ecc097-1c88-4f8c-928c-708b9a628a72-host-slash\") pod \"iptables-alerter-s2wpv\" (UID: \"82ecc097-1c88-4f8c-928c-708b9a628a72\") " pod="openshift-network-operator/iptables-alerter-s2wpv" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380845 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380868 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-netd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380900 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-hostroot\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380910 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-cni-netd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380929 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-sys\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.380964 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-var-lib-kubelet\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381000 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381029 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381028 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381075 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381090 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381117 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e7f16d-ec14-40bd-908f-272016e1464f-metrics-client-ca\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381122 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381144 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vx5z\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381162 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-hostroot\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381186 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381172 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052e1a61-3220-4a91-83c5-a08bb829f358-tmp-dir\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381222 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysconfig\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381233 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-slash\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381240 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381253 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052e1a61-3220-4a91-83c5-a08bb829f358-tmp-dir\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381265 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381268 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-run-multus-certs\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381339 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-etc-sysconfig\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381374 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e7f16d-ec14-40bd-908f-272016e1464f-metrics-client-ca\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.381336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381409 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381431 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.381446 6727 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381545 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-dbus\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381568 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381582 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-lib-modules\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381600 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-node-log\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381615 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381629 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381654 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2e7f16d-ec14-40bd-908f-272016e1464f-tmp\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381668 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-registration-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381682 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-conf-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381696 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381714 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381730 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/052e1a61-3220-4a91-83c5-a08bb829f358-config-volume\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381746 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381761 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381777 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-system-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381790 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-socket-dir-parent\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381805 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-kubelet\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381822 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381838 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381855 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-systemd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381872 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381887 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381902 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381918 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.381948 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382087 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382129 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a166dd5f-2e0c-45a0-8dc6-692741a8e1b3-dbus\") pod \"global-pull-secret-syncer-g7zbh\" (UID: \"a166dd5f-2e0c-45a0-8dc6-692741a8e1b3\") " pod="kube-system/global-pull-secret-syncer-g7zbh" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.382150 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:10.882139727 +0000 UTC m=+1.697154096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382285 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-tls\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382356 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958a8aee-281c-45d9-8c87-ee2ce835fbee-lib-modules\") pod \"tuned-sv5jl\" (UID: \"958a8aee-281c-45d9-8c87-ee2ce835fbee\") " pod="openshift-cluster-node-tuning-operator/tuned-sv5jl" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382380 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-node-log\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.382413 6727 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.382450 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:10.882442105 +0000 UTC m=+1.697456474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"service-ca" not registered Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382526 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-system-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382560 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2b3ee2-f439-49ca-a106-63e231dc232f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnnvc\" (UID: \"ad2b3ee2-f439-49ca-a106-63e231dc232f\") " pod="openshift-multus/multus-additional-cni-plugins-wnnvc" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382564 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1925581-bb98-4bd2-8c66-5cafa370dfc9-registration-dir\") pod \"aws-ebs-csi-driver-node-dmgrh\" (UID: \"b1925581-bb98-4bd2-8c66-5cafa370dfc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmgrh" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382569 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-conf-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382591 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382638 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2e7f16d-ec14-40bd-908f-272016e1464f-tmp\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382638 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382692 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-cni-dir\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382696 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e7f16d-ec14-40bd-908f-272016e1464f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382733 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-multus-socket-dir-parent\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382739 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382837 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382869 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382886 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b626d907-7b4b-4437-9e69-a82333ebf268-host-var-lib-kubelet\") pod \"multus-p2ppd\" (UID: \"b626d907-7b4b-4437-9e69-a82333ebf268\") " pod="openshift-multus/multus-p2ppd" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382896 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.382906 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/052e1a61-3220-4a91-83c5-a08bb829f358-config-volume\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.383009 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.383115 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.383191 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97b652f-9d17-4078-90ea-ac9bd9820384-run-systemd\") pod \"ovnkube-node-jq9k9\" (UID: \"d97b652f-9d17-4078-90ea-ac9bd9820384\") " pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:10.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.384352 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.388538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.388516 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4f5z\" (UniqueName: \"kubernetes.io/projected/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-kube-api-access-z4f5z\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.388856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.388838 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kdf\" (UniqueName: \"kubernetes.io/projected/0d06e6e9-51c0-403a-8ea0-ecc9de005803-kube-api-access-p7kdf\") pod \"monitoring-plugin-77f4f76594-wqp22\" (UID: \"0d06e6e9-51c0-403a-8ea0-ecc9de005803\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:28:10.389634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.389616 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7qm\" (UniqueName: \"kubernetes.io/projected/a95a2e68-c596-4c2e-b1bd-20d31bb31919-kube-api-access-2t7qm\") pod \"openshift-state-metrics-7bd7489957-bp6vl\" (UID: \"a95a2e68-c596-4c2e-b1bd-20d31bb31919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bp6vl" Oct 31 12:28:10.389768 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.389755 6727 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:28:10.389810 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.389770 6727 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:28:10.389810 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.389778 6727 projected.go:196] Error preparing data for projected volume kube-api-access-ckfkv for pod openshift-console/console-57dbb8df47-wq5vh: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:10.389876 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.389818 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:10.889808864 +0000 UTC m=+1.704823232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ckfkv" (UniqueName: "kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:10.390221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.390205 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79d28\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\") pod \"alertmanager-main-0\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:10.390319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.390303 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbr8s\" (UniqueName: \"kubernetes.io/projected/d2e7f16d-ec14-40bd-908f-272016e1464f-kube-api-access-jbr8s\") pod \"thanos-querier-577cf77cd5-p495x\" (UID: \"d2e7f16d-ec14-40bd-908f-272016e1464f\") " pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:10.390609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.390595 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdpk5\" (UniqueName: \"kubernetes.io/projected/052e1a61-3220-4a91-83c5-a08bb829f358-kube-api-access-qdpk5\") pod \"dns-default-4jx6k\" (UID: \"052e1a61-3220-4a91-83c5-a08bb829f358\") " pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:10.390832 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.390818 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnw6\" (UniqueName: \"kubernetes.io/projected/7c0a973f-9d43-4bec-a134-eda7d2b7a052-kube-api-access-wxnw6\") pod \"kube-state-metrics-5dc8d7d89b-gbwhw\" (UID: \"7c0a973f-9d43-4bec-a134-eda7d2b7a052\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-gbwhw" Oct 31 12:28:10.391082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.391066 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vx5z\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\") pod \"prometheus-k8s-0\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:10.482690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.482663 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-sys\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.482798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.482759 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-sys\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.482798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.482767 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-root\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.482902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.482803 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-wtmp\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.482902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.482859 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-root\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.482995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.482950 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105-node-exporter-wtmp\") pod \"node-exporter-67z7l\" (UID: \"d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105\") " pod="openshift-monitoring/node-exporter-67z7l" Oct 31 12:28:10.886166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.886138 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.886271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.886181 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.886271 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886254 6727 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:10.886346 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886272 6727 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:10.886346 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886304 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:11.886292799 +0000 UTC m=+2.701307168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:10.886346 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886324 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:11.886310959 +0000 UTC m=+2.701325327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:10.886346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.886345 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.886542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.886405 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.886542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.886458 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.886542 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886490 6727 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:10.886649 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886551 6727 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:28:10.886649 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886510 6727 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:28:10.886649 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886585 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:11.88656782 +0000 UTC m=+2.701582198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:10.886649 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886606 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:11.88659297 +0000 UTC m=+2.701607358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"service-ca" not registered Oct 31 12:28:10.886649 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.886622 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:11.886614148 +0000 UTC m=+2.701628518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-config" not registered Oct 31 12:28:10.987360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:10.987330 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:10.987502 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.987483 6727 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:28:10.987563 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.987504 6727 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:28:10.987563 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.987516 6727 projected.go:196] Error preparing data for projected volume kube-api-access-ckfkv for pod openshift-console/console-57dbb8df47-wq5vh: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:10.987563 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:10.987557 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:11.987546804 +0000 UTC m=+2.802561236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckfkv" (UniqueName: "kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:11.335666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.335644 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4jx6k" Oct 31 12:28:11.896555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.896517 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:11.896730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.896562 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:11.896730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.896603 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:11.896730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.896645 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:11.896730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.896672 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:11.896730 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896680 6727 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:28:11.896730 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896689 6727 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896757 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:13.896737265 +0000 UTC m=+4.711751659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"service-ca" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896759 6727 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896777 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:13.896767416 +0000 UTC m=+4.711781802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896779 6727 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896787 6727 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896811 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:13.896797008 +0000 UTC m=+4.711811377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896827 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:13.896820226 +0000 UTC m=+4.711834595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-config" not registered Oct 31 12:28:11.896993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.896899 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:13.896881115 +0000 UTC m=+4.711895515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:11.990999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.990977 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:11.997824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:11.997803 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:11.997916 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.997905 6727 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:28:11.997956 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.997920 6727 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:28:11.997956 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.997931 6727 projected.go:196] Error preparing data for projected volume kube-api-access-ckfkv for pod openshift-console/console-57dbb8df47-wq5vh: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:11.998012 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:11.997981 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:13.997967578 +0000 UTC m=+4.812981963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckfkv" (UniqueName: "kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:12.847806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:12.847781 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-577cf77cd5-p495x" Oct 31 12:28:13.546756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:13.546730 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:13.914648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:13.914614 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:13.914672 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:13.914697 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914750 6727 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:13.914775 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914794 6727 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914805 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:17.914791817 +0000 UTC m=+8.729806209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914843 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:17.914829977 +0000 UTC m=+8.729844386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:13.914837 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914858 6727 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914868 6727 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914874 6727 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914903 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:17.914890203 +0000 UTC m=+8.729904586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914934 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:17.914924908 +0000 UTC m=+8.729939299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-config" not registered Oct 31 12:28:13.915132 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:13.914949 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:17.914941292 +0000 UTC m=+8.729955662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"service-ca" not registered Oct 31 12:28:14.016050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:14.016013 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:14.016185 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:14.016164 6727 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:28:14.016185 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:14.016182 6727 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:28:14.016254 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:14.016193 6727 projected.go:196] Error preparing data for projected volume kube-api-access-ckfkv for pod openshift-console/console-57dbb8df47-wq5vh: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:14.016254 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:14.016233 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:18.016220063 +0000 UTC m=+8.831234440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckfkv" (UniqueName: "kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:14.541655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:14.541627 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:14.541948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:14.541931 6727 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:28:15.374076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:15.374045 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:28:17.563723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:17.563680 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:17.691827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:17.691786 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:17.963062 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:17.963021 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:17.963062 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:17.963069 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:17.963088 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:17.963137 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:17.963156 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963166 6727 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963201 6727 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963229 6727 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963263 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:25.963242815 +0000 UTC m=+16.778257191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963274 6727 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963289 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:25.963272567 +0000 UTC m=+16.778286936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"service-ca" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963287 6727 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963307 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:25.963297285 +0000 UTC m=+16.778311654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-config" not registered Oct 31 12:28:17.963337 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963328 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:25.963316605 +0000 UTC m=+16.778331007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:17.963857 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:17.963351 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:25.963339243 +0000 UTC m=+16.778353611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:18.063847 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:18.063813 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:18.063993 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:18.063975 6727 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:28:18.064040 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:18.063998 6727 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:28:18.064040 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:18.064010 6727 projected.go:196] Error preparing data for projected volume kube-api-access-ckfkv for pod openshift-console/console-57dbb8df47-wq5vh: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:18.064100 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:18.064065 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:26.064053598 +0000 UTC m=+16.879067966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckfkv" (UniqueName: "kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:18.672221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:18.672192 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:18.843230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:18.843201 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:19.500774 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.500752 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wqp22" Oct 31 12:28:19.702183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.702116 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:28:19.702548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.702399 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="alertmanager" containerID="cri-o://8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8" gracePeriod=120 Oct 31 12:28:19.702548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.702492 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-web" containerID="cri-o://4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4" gracePeriod=120 Oct 31 12:28:19.702548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.702484 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-metric" containerID="cri-o://79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb" gracePeriod=120 Oct 31 12:28:19.702548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.702502 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="config-reloader" containerID="cri-o://475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831" gracePeriod=120 Oct 31 12:28:19.702765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.702562 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="prom-label-proxy" containerID="cri-o://a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb" gracePeriod=120 Oct 31 12:28:19.702765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:19.702601 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy" containerID="cri-o://8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec" gracePeriod=120 Oct 31 12:28:20.399540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399511 6727 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb" exitCode=0 Oct 31 12:28:20.399540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399536 6727 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec" exitCode=0 Oct 31 12:28:20.399540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399543 6727 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831" exitCode=0 Oct 31 12:28:20.399731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399549 6727 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8" exitCode=0 Oct 31 12:28:20.399731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399580 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb"} Oct 31 12:28:20.399731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399667 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec"} Oct 31 12:28:20.399731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399695 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831"} Oct 31 12:28:20.399731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.399719 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8"} Oct 31 12:28:20.980205 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:20.980180 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.090380 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090358 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090400 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090437 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090484 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090512 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090551 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090629 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090661 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090712 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.090757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090743 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.091000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090771 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:21.091000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090818 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.091000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090862 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.091000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.090926 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5ac5a328-04f5-4857-bcc8-e081c18907d3\" (UID: \"5ac5a328-04f5-4857-bcc8-e081c18907d3\") " Oct 31 12:28:21.091234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.091209 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:21.091291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.091234 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-metrics-client-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.091662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.091633 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:28:21.094681 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.094657 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:28:21.094796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.094699 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:21.094796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.094769 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:28:21.094912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.094842 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "kube-api-access-79d28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:28:21.094912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.094855 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:21.095008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.094992 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:21.095114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.095100 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:21.095275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.095262 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:21.098552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.098529 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:21.105420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.105392 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config" pod "5ac5a328-04f5-4857-bcc8-e081c18907d3" (UID: "5ac5a328-04f5-4857-bcc8-e081c18907d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:21.191874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191854 6727 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-cluster-tls-config\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.191874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191875 6727 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191885 6727 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191894 6727 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-config-out\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191902 6727 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-tls-assets\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191911 6727 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ac5a328-04f5-4857-bcc8-e081c18907d3-alertmanager-main-db\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191921 6727 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191930 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79d28\" (UniqueName: \"kubernetes.io/projected/5ac5a328-04f5-4857-bcc8-e081c18907d3-kube-api-access-79d28\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191938 6727 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-main-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191947 6727 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191956 6727 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-web-config\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.192004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.191965 6727 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ac5a328-04f5-4857-bcc8-e081c18907d3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:21.413106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.412738 6727 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb" exitCode=0 Oct 31 12:28:21.413106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.412770 6727 generic.go:356] "Generic (PLEG): container finished" podID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerID="4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4" exitCode=0 Oct 31 12:28:21.413106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.412846 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb"} Oct 31 12:28:21.413106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.412917 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4"} Oct 31 12:28:21.413106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.412956 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ac5a328-04f5-4857-bcc8-e081c18907d3","Type":"ContainerDied","Data":"05afc9f1f06d7ed71042fbdbcc30ea7c79e1e53d369ff360e68405238fd537d3"} Oct 31 12:28:21.413106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.412992 6727 scope.go:122] "RemoveContainer" containerID="a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb" Oct 31 12:28:21.413396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.413142 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.421445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.421425 6727 scope.go:122] "RemoveContainer" containerID="79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb" Oct 31 12:28:21.428490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.428472 6727 scope.go:122] "RemoveContainer" containerID="8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec" Oct 31 12:28:21.435828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.435805 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:28:21.436316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.436293 6727 scope.go:122] "RemoveContainer" containerID="4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4" Oct 31 12:28:21.438917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.438895 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:28:21.443395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.443373 6727 scope.go:122] "RemoveContainer" containerID="475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831" Oct 31 12:28:21.451457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.451438 6727 scope.go:122] "RemoveContainer" containerID="8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8" Oct 31 12:28:21.459001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.458963 6727 scope.go:122] "RemoveContainer" containerID="e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937" Oct 31 12:28:21.466906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.466885 6727 scope.go:122] "RemoveContainer" containerID="a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb" Oct 31 12:28:21.467162 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:21.467144 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb\": container with ID starting with a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb not found: ID does not exist" containerID="a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb" Oct 31 12:28:21.467231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.467172 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb"} err="failed to get container status \"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb\": rpc error: code = NotFound desc = could not find container \"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb\": container with ID starting with a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb not found: ID does not exist" Oct 31 12:28:21.467231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.467206 6727 scope.go:122] "RemoveContainer" containerID="79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb" Oct 31 12:28:21.467421 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:21.467403 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb\": container with ID starting with 79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb not found: ID does not exist" containerID="79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb" Oct 31 12:28:21.467517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.467427 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb"} err="failed to get container status \"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb\": rpc error: code = NotFound desc = could not find container \"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb\": container with ID starting with 79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb not found: ID does not exist" Oct 31 12:28:21.467517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.467440 6727 scope.go:122] "RemoveContainer" containerID="8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec" Oct 31 12:28:21.467756 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:21.467731 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec\": container with ID starting with 8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec not found: ID does not exist" containerID="8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec" Oct 31 12:28:21.467805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.467765 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec"} err="failed to get container status \"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec\": rpc error: code = NotFound desc = could not find container \"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec\": container with ID starting with 8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec not found: ID does not exist" Oct 31 12:28:21.467805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.467787 6727 scope.go:122] "RemoveContainer" containerID="4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4" Oct 31 12:28:21.468041 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:21.468024 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4\": container with ID starting with 4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4 not found: ID does not exist" containerID="4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4" Oct 31 12:28:21.468087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468046 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4"} err="failed to get container status \"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4\": rpc error: code = NotFound desc = could not find container \"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4\": container with ID starting with 4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4 not found: ID does not exist" Oct 31 12:28:21.468087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468060 6727 scope.go:122] "RemoveContainer" containerID="475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831" Oct 31 12:28:21.468289 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:21.468272 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831\": container with ID starting with 475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831 not found: ID does not exist" containerID="475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831" Oct 31 12:28:21.468328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468294 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831"} err="failed to get container status \"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831\": rpc error: code = NotFound desc = could not find container \"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831\": container with ID starting with 475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831 not found: ID does not exist" Oct 31 12:28:21.468328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468309 6727 scope.go:122] "RemoveContainer" containerID="8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8" Oct 31 12:28:21.468514 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:21.468499 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8\": container with ID starting with 8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8 not found: ID does not exist" containerID="8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8" Oct 31 12:28:21.468514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468517 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8"} err="failed to get container status \"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8\": rpc error: code = NotFound desc = could not find container \"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8\": container with ID starting with 8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8 not found: ID does not exist" Oct 31 12:28:21.468630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468528 6727 scope.go:122] "RemoveContainer" containerID="e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937" Oct 31 12:28:21.468768 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:21.468754 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937\": container with ID starting with e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937 not found: ID does not exist" containerID="e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937" Oct 31 12:28:21.468809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468772 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937"} err="failed to get container status \"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937\": rpc error: code = NotFound desc = could not find container \"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937\": container with ID starting with e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937 not found: ID does not exist" Oct 31 12:28:21.468809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468783 6727 scope.go:122] "RemoveContainer" containerID="a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb" Oct 31 12:28:21.469005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.468990 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb"} err="failed to get container status \"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb\": rpc error: code = NotFound desc = could not find container \"a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb\": container with ID starting with a92022245fea77c3334aa35ea5bf09617a3774cef73f807c5b9e74335a50ffbb not found: ID does not exist" Oct 31 12:28:21.469059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469004 6727 scope.go:122] "RemoveContainer" containerID="79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb" Oct 31 12:28:21.469240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469218 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb"} err="failed to get container status \"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb\": rpc error: code = NotFound desc = could not find container \"79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb\": container with ID starting with 79a766c380906aa6d3ec7db8eb00d16b7d9ce202d954abcc3bf5d36407d1a4bb not found: ID does not exist" Oct 31 12:28:21.469291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469242 6727 scope.go:122] "RemoveContainer" containerID="8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec" Oct 31 12:28:21.469473 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469443 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec"} err="failed to get container status \"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec\": rpc error: code = NotFound desc = could not find container \"8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec\": container with ID starting with 8299a3b853f2568a96d29d6771e3b7a26802e96ffeef8eb8d59f92d9b9728aec not found: ID does not exist" Oct 31 12:28:21.469530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469483 6727 scope.go:122] "RemoveContainer" containerID="4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4" Oct 31 12:28:21.469723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469701 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4"} err="failed to get container status \"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4\": rpc error: code = NotFound desc = could not find container \"4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4\": container with ID starting with 4e8f7fed7d123d1b2458caa888b2375fbd3848a448d647be3a90b98990dcc5d4 not found: ID does not exist" Oct 31 12:28:21.469769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469725 6727 scope.go:122] "RemoveContainer" containerID="475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831" Oct 31 12:28:21.469924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469909 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831"} err="failed to get container status \"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831\": rpc error: code = NotFound desc = could not find container \"475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831\": container with ID starting with 475ff9b68b96cea094e5651fe545afce9322e15629ab4a49513eccdbc6979831 not found: ID does not exist" Oct 31 12:28:21.469965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.469924 6727 scope.go:122] "RemoveContainer" containerID="8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8" Oct 31 12:28:21.470124 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.470104 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8"} err="failed to get container status \"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8\": rpc error: code = NotFound desc = could not find container \"8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8\": container with ID starting with 8206790d2a84cbd61687c2da4893fc64fc029bd8d1432519c63bfbf5bca12ef8 not found: ID does not exist" Oct 31 12:28:21.470167 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.470126 6727 scope.go:122] "RemoveContainer" containerID="e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937" Oct 31 12:28:21.470373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.470356 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937"} err="failed to get container status \"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937\": rpc error: code = NotFound desc = could not find container \"e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937\": container with ID starting with e450eada45a238e0d4e7f9b24c73fd46207bf0c30c2bddafb6921442d0a04937 not found: ID does not exist" Oct 31 12:28:21.472775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.472755 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:28:21.473038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473026 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-metric" Oct 31 12:28:21.473081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473039 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-metric" Oct 31 12:28:21.473081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473050 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy" Oct 31 12:28:21.473081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473054 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy" Oct 31 12:28:21.473081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473066 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="config-reloader" Oct 31 12:28:21.473081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473070 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="config-reloader" Oct 31 12:28:21.473081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473079 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-web" Oct 31 12:28:21.473081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473084 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-web" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473092 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="prom-label-proxy" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473097 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="prom-label-proxy" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473108 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="init-config-reloader" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473112 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="init-config-reloader" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473121 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="alertmanager" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473125 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="alertmanager" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473170 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-metric" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473178 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473183 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="config-reloader" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473190 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="kube-rbac-proxy-web" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473195 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="prom-label-proxy" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473201 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="init-config-reloader" Oct 31 12:28:21.473303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.473207 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" containerName="alertmanager" Oct 31 12:28:21.511061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.511029 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:28:21.511168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.511150 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.515114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515093 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 12:28:21.515541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515364 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 12:28:21.515541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515375 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 12:28:21.515541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515412 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 12:28:21.515541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515420 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 12:28:21.515541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515400 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 12:28:21.515541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515477 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mjk8n\"" Oct 31 12:28:21.515541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515495 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 12:28:21.515827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.515560 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 12:28:21.517215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.517189 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 12:28:21.595786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.595765 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f94000a6-09f2-4acb-8538-99d495b050f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.595883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.595794 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.595883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.595814 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.595883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.595868 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f94000a6-09f2-4acb-8538-99d495b050f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.595990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.595972 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f94000a6-09f2-4acb-8538-99d495b050f1-config-out\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.595998 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.596021 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.596063 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-web-config\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.596121 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f94000a6-09f2-4acb-8538-99d495b050f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.596153 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.596178 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.596223 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvw8\" (UniqueName: \"kubernetes.io/projected/f94000a6-09f2-4acb-8538-99d495b050f1-kube-api-access-lpvw8\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.596272 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.596259 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94000a6-09f2-4acb-8538-99d495b050f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697504 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f94000a6-09f2-4acb-8538-99d495b050f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697532 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697549 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697573 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f94000a6-09f2-4acb-8538-99d495b050f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697764 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f94000a6-09f2-4acb-8538-99d495b050f1-config-out\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697795 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697827 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697855 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-web-config\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697895 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f94000a6-09f2-4acb-8538-99d495b050f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.697908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697898 6727 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:28:21.698114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697897 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f94000a6-09f2-4acb-8538-99d495b050f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.698114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697947 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.698114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697977 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.698114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.697996 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvw8\" (UniqueName: \"kubernetes.io/projected/f94000a6-09f2-4acb-8538-99d495b050f1-kube-api-access-lpvw8\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.698114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.698025 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94000a6-09f2-4acb-8538-99d495b050f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.698368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.698300 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f94000a6-09f2-4acb-8538-99d495b050f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.698915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.698891 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94000a6-09f2-4acb-8538-99d495b050f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.700680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.700656 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f94000a6-09f2-4acb-8538-99d495b050f1-config-out\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.700807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.700786 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.700910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.700893 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.701107 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.701084 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-web-config\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.701203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.701145 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.701247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.701208 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.701247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.701222 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.701409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.701388 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f94000a6-09f2-4acb-8538-99d495b050f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.702632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.702613 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f94000a6-09f2-4acb-8538-99d495b050f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.710164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.710147 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvw8\" (UniqueName: \"kubernetes.io/projected/f94000a6-09f2-4acb-8538-99d495b050f1-kube-api-access-lpvw8\") pod \"alertmanager-main-0\" (UID: \"f94000a6-09f2-4acb-8538-99d495b050f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:21.914079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:21.914056 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:28:22.235911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:22.235691 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:28:22.239260 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:22.239224 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94000a6_09f2_4acb_8538_99d495b050f1.slice/crio-8cecc248c8974581bbfda2f8d27b625a2f017f19ffb22758d899009ddbb0b8c6 WatchSource:0}: Error finding container 8cecc248c8974581bbfda2f8d27b625a2f017f19ffb22758d899009ddbb0b8c6: Status 404 returned error can't find the container with id 8cecc248c8974581bbfda2f8d27b625a2f017f19ffb22758d899009ddbb0b8c6 Oct 31 12:28:22.380841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:22.380816 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:22.417924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:22.417897 6727 generic.go:356] "Generic (PLEG): container finished" podID="f94000a6-09f2-4acb-8538-99d495b050f1" containerID="8a90905a19dd382c4fa0034f676df6e74e3985d7097f0c35cc0fa34707fde965" exitCode=0 Oct 31 12:28:22.418009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:22.417990 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerDied","Data":"8a90905a19dd382c4fa0034f676df6e74e3985d7097f0c35cc0fa34707fde965"} Oct 31 12:28:22.418098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:22.418081 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerStarted","Data":"8cecc248c8974581bbfda2f8d27b625a2f017f19ffb22758d899009ddbb0b8c6"} Oct 31 12:28:22.519535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:22.519513 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:23.297324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.297296 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac5a328-04f5-4857-bcc8-e081c18907d3" path="/var/lib/kubelet/pods/5ac5a328-04f5-4857-bcc8-e081c18907d3/volumes" Oct 31 12:28:23.425563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.425530 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerStarted","Data":"ae8bccf60ea617c1106972b3464bab3d4ad92a3e63160be783988e0f2abf0237"} Oct 31 12:28:23.425675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.425612 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerStarted","Data":"e411ca2a6eea53a8ac3e62479d5c6e36182ae9f32b83088118913f45b8789bc9"} Oct 31 12:28:23.425675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.425653 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerStarted","Data":"d36ba030e4b6e8ad6379217d618f82f00cd81abaf9c7f12344ff9630c681c738"} Oct 31 12:28:23.425752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.425697 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerStarted","Data":"53acf990c1ea83014c6a83dabe3af7788c2f5a22bcc86cb356a90fe8982f470c"} Oct 31 12:28:23.425752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.425733 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerStarted","Data":"93d525aaea0f9dc5b7d121c90131130da6fa0696f5c0b18d568177fe1c7533c0"} Oct 31 12:28:23.425807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.425769 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f94000a6-09f2-4acb-8538-99d495b050f1","Type":"ContainerStarted","Data":"b9bcdc8298270234b623147ae99bcdd3d29d6d133c7a784b9cd32ac68177d1b3"} Oct 31 12:28:23.457688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:23.457642 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.444143464 podStartE2EDuration="2.457628667s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.485203ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:28:23.457062913 +0000 UTC m=+14.272077334" watchObservedRunningTime="2026-10-31 12:28:23.457628667 +0000 UTC m=+14.272643058" Oct 31 12:28:24.077480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.077205 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:28:24.077655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.077549 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="prometheus" containerID="cri-o://1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" gracePeriod=600 Oct 31 12:28:24.077800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.077770 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-web" containerID="cri-o://b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" gracePeriod=600 Oct 31 12:28:24.077888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.077806 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy" containerID="cri-o://1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" gracePeriod=600 Oct 31 12:28:24.077888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.077819 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="thanos-sidecar" containerID="cri-o://b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" gracePeriod=600 Oct 31 12:28:24.077984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.077806 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="config-reloader" containerID="cri-o://68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" gracePeriod=600 Oct 31 12:28:24.078731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.077770 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-thanos" containerID="cri-o://61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" gracePeriod=600 Oct 31 12:28:24.346350 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.346301 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.432826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432792 6727 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" exitCode=0 Oct 31 12:28:24.432826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432817 6727 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" exitCode=0 Oct 31 12:28:24.432826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432824 6727 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" exitCode=0 Oct 31 12:28:24.432826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432830 6727 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" exitCode=0 Oct 31 12:28:24.433034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432836 6727 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" exitCode=0 Oct 31 12:28:24.433034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432843 6727 generic.go:356] "Generic (PLEG): container finished" podID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" exitCode=0 Oct 31 12:28:24.433034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432872 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.433034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432872 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} Oct 31 12:28:24.433034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432952 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} Oct 31 12:28:24.433034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.432996 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} Oct 31 12:28:24.433034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.433016 6727 scope.go:122] "RemoveContainer" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.433284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.433037 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} Oct 31 12:28:24.433284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.433078 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} Oct 31 12:28:24.433284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.433119 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} Oct 31 12:28:24.433284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.433158 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d26075f-7f96-4d14-bf9e-85ec07189e38","Type":"ContainerDied","Data":"70a4a90f0770abbc2d635d6b7a7987756c9c0d9357e3eec443ef0a7ea96a15a9"} Oct 31 12:28:24.442096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.442056 6727 scope.go:122] "RemoveContainer" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.450494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.450457 6727 scope.go:122] "RemoveContainer" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.456882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.456866 6727 scope.go:122] "RemoveContainer" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.463485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.463449 6727 scope.go:122] "RemoveContainer" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.470002 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.469984 6727 scope.go:122] "RemoveContainer" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.476515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.476493 6727 scope.go:122] "RemoveContainer" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.482512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.482492 6727 scope.go:122] "RemoveContainer" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.482777 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:24.482758 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": container with ID starting with 61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb not found: ID does not exist" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.482837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.482785 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} err="failed to get container status \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": rpc error: code = NotFound desc = could not find container \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": container with ID starting with 61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb not found: ID does not exist" Oct 31 12:28:24.482837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.482804 6727 scope.go:122] "RemoveContainer" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.483072 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:24.483055 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": container with ID starting with 1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848 not found: ID does not exist" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.483115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483078 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} err="failed to get container status \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": rpc error: code = NotFound desc = could not find container \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": container with ID starting with 1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848 not found: ID does not exist" Oct 31 12:28:24.483115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483094 6727 scope.go:122] "RemoveContainer" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.483307 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:24.483287 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": container with ID starting with b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191 not found: ID does not exist" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.483346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483312 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} err="failed to get container status \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": rpc error: code = NotFound desc = could not find container \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": container with ID starting with b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191 not found: ID does not exist" Oct 31 12:28:24.483346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483325 6727 scope.go:122] "RemoveContainer" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.483553 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:24.483537 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": container with ID starting with b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e not found: ID does not exist" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.483589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483560 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} err="failed to get container status \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": rpc error: code = NotFound desc = could not find container \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": container with ID starting with b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e not found: ID does not exist" Oct 31 12:28:24.483589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483574 6727 scope.go:122] "RemoveContainer" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.483805 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:24.483791 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": container with ID starting with 68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6 not found: ID does not exist" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.483840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483807 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} err="failed to get container status \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": rpc error: code = NotFound desc = could not find container \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": container with ID starting with 68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6 not found: ID does not exist" Oct 31 12:28:24.483840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.483820 6727 scope.go:122] "RemoveContainer" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.484013 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:24.483996 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": container with ID starting with 1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a not found: ID does not exist" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.484059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484018 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} err="failed to get container status \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": rpc error: code = NotFound desc = could not find container \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": container with ID starting with 1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a not found: ID does not exist" Oct 31 12:28:24.484059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484031 6727 scope.go:122] "RemoveContainer" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.484230 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:24.484215 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": container with ID starting with 014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244 not found: ID does not exist" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.484271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484233 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} err="failed to get container status \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": rpc error: code = NotFound desc = could not find container \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": container with ID starting with 014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244 not found: ID does not exist" Oct 31 12:28:24.484271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484244 6727 scope.go:122] "RemoveContainer" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.484444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484429 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} err="failed to get container status \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": rpc error: code = NotFound desc = could not find container \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": container with ID starting with 61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb not found: ID does not exist" Oct 31 12:28:24.484589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484444 6727 scope.go:122] "RemoveContainer" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.484704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484687 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} err="failed to get container status \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": rpc error: code = NotFound desc = could not find container \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": container with ID starting with 1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848 not found: ID does not exist" Oct 31 12:28:24.484746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484705 6727 scope.go:122] "RemoveContainer" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.484910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484889 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} err="failed to get container status \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": rpc error: code = NotFound desc = could not find container \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": container with ID starting with b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191 not found: ID does not exist" Oct 31 12:28:24.484948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.484909 6727 scope.go:122] "RemoveContainer" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.485090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485074 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} err="failed to get container status \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": rpc error: code = NotFound desc = could not find container \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": container with ID starting with b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e not found: ID does not exist" Oct 31 12:28:24.485130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485090 6727 scope.go:122] "RemoveContainer" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.485306 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485287 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} err="failed to get container status \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": rpc error: code = NotFound desc = could not find container \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": container with ID starting with 68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6 not found: ID does not exist" Oct 31 12:28:24.485354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485306 6727 scope.go:122] "RemoveContainer" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.485524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485510 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} err="failed to get container status \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": rpc error: code = NotFound desc = could not find container \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": container with ID starting with 1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a not found: ID does not exist" Oct 31 12:28:24.485524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485524 6727 scope.go:122] "RemoveContainer" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.485735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485717 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} err="failed to get container status \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": rpc error: code = NotFound desc = could not find container \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": container with ID starting with 014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244 not found: ID does not exist" Oct 31 12:28:24.485777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485734 6727 scope.go:122] "RemoveContainer" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.485920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485905 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} err="failed to get container status \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": rpc error: code = NotFound desc = could not find container \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": container with ID starting with 61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb not found: ID does not exist" Oct 31 12:28:24.485957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.485920 6727 scope.go:122] "RemoveContainer" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.486147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486130 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} err="failed to get container status \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": rpc error: code = NotFound desc = could not find container \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": container with ID starting with 1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848 not found: ID does not exist" Oct 31 12:28:24.486192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486148 6727 scope.go:122] "RemoveContainer" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.486348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486333 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} err="failed to get container status \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": rpc error: code = NotFound desc = could not find container \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": container with ID starting with b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191 not found: ID does not exist" Oct 31 12:28:24.486348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486347 6727 scope.go:122] "RemoveContainer" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.486582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486564 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} err="failed to get container status \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": rpc error: code = NotFound desc = could not find container \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": container with ID starting with b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e not found: ID does not exist" Oct 31 12:28:24.486627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486582 6727 scope.go:122] "RemoveContainer" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.486777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486762 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} err="failed to get container status \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": rpc error: code = NotFound desc = could not find container \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": container with ID starting with 68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6 not found: ID does not exist" Oct 31 12:28:24.486823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486776 6727 scope.go:122] "RemoveContainer" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.486967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486950 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} err="failed to get container status \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": rpc error: code = NotFound desc = could not find container \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": container with ID starting with 1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a not found: ID does not exist" Oct 31 12:28:24.487007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.486967 6727 scope.go:122] "RemoveContainer" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.487162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487148 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} err="failed to get container status \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": rpc error: code = NotFound desc = could not find container \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": container with ID starting with 014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244 not found: ID does not exist" Oct 31 12:28:24.487198 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487161 6727 scope.go:122] "RemoveContainer" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.487349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487335 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} err="failed to get container status \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": rpc error: code = NotFound desc = could not find container \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": container with ID starting with 61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb not found: ID does not exist" Oct 31 12:28:24.487349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487348 6727 scope.go:122] "RemoveContainer" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.487546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487530 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} err="failed to get container status \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": rpc error: code = NotFound desc = could not find container \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": container with ID starting with 1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848 not found: ID does not exist" Oct 31 12:28:24.487546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487545 6727 scope.go:122] "RemoveContainer" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.487709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487688 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} err="failed to get container status \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": rpc error: code = NotFound desc = could not find container \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": container with ID starting with b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191 not found: ID does not exist" Oct 31 12:28:24.487760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487711 6727 scope.go:122] "RemoveContainer" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.487897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487877 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} err="failed to get container status \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": rpc error: code = NotFound desc = could not find container \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": container with ID starting with b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e not found: ID does not exist" Oct 31 12:28:24.487964 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.487898 6727 scope.go:122] "RemoveContainer" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.488093 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488078 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} err="failed to get container status \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": rpc error: code = NotFound desc = could not find container \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": container with ID starting with 68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6 not found: ID does not exist" Oct 31 12:28:24.488093 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488092 6727 scope.go:122] "RemoveContainer" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.488294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488277 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} err="failed to get container status \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": rpc error: code = NotFound desc = could not find container \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": container with ID starting with 1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a not found: ID does not exist" Oct 31 12:28:24.488336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488293 6727 scope.go:122] "RemoveContainer" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.488552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488535 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} err="failed to get container status \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": rpc error: code = NotFound desc = could not find container \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": container with ID starting with 014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244 not found: ID does not exist" Oct 31 12:28:24.488617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488554 6727 scope.go:122] "RemoveContainer" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.488793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488772 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} err="failed to get container status \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": rpc error: code = NotFound desc = could not find container \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": container with ID starting with 61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb not found: ID does not exist" Oct 31 12:28:24.488793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488790 6727 scope.go:122] "RemoveContainer" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.488984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488967 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} err="failed to get container status \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": rpc error: code = NotFound desc = could not find container \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": container with ID starting with 1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848 not found: ID does not exist" Oct 31 12:28:24.489027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.488984 6727 scope.go:122] "RemoveContainer" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.489175 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489159 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} err="failed to get container status \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": rpc error: code = NotFound desc = could not find container \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": container with ID starting with b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191 not found: ID does not exist" Oct 31 12:28:24.489221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489175 6727 scope.go:122] "RemoveContainer" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.489353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489337 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} err="failed to get container status \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": rpc error: code = NotFound desc = could not find container \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": container with ID starting with b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e not found: ID does not exist" Oct 31 12:28:24.489353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489351 6727 scope.go:122] "RemoveContainer" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.489580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489563 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} err="failed to get container status \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": rpc error: code = NotFound desc = could not find container \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": container with ID starting with 68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6 not found: ID does not exist" Oct 31 12:28:24.489633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489580 6727 scope.go:122] "RemoveContainer" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.489780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489765 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} err="failed to get container status \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": rpc error: code = NotFound desc = could not find container \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": container with ID starting with 1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a not found: ID does not exist" Oct 31 12:28:24.489822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489780 6727 scope.go:122] "RemoveContainer" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.489990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489974 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} err="failed to get container status \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": rpc error: code = NotFound desc = could not find container \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": container with ID starting with 014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244 not found: ID does not exist" Oct 31 12:28:24.490028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.489990 6727 scope.go:122] "RemoveContainer" containerID="61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb" Oct 31 12:28:24.490188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490173 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb"} err="failed to get container status \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": rpc error: code = NotFound desc = could not find container \"61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb\": container with ID starting with 61dc26e5bf20502cba4c4a65300ccfa144204b935bb22c42e9f2ea904229d8fb not found: ID does not exist" Oct 31 12:28:24.490231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490189 6727 scope.go:122] "RemoveContainer" containerID="1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848" Oct 31 12:28:24.490378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490364 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848"} err="failed to get container status \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": rpc error: code = NotFound desc = could not find container \"1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848\": container with ID starting with 1f7be2767518f17d5060285f2268d03fa3ab6f9b022cecb2ac7c3ae9da193848 not found: ID does not exist" Oct 31 12:28:24.490427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490378 6727 scope.go:122] "RemoveContainer" containerID="b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191" Oct 31 12:28:24.490587 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490572 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191"} err="failed to get container status \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": rpc error: code = NotFound desc = could not find container \"b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191\": container with ID starting with b3c2bb0edf9065a8e70bf0a56a7945b7ff6df8836bc2e4f80ac312ddc581b191 not found: ID does not exist" Oct 31 12:28:24.490636 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490586 6727 scope.go:122] "RemoveContainer" containerID="b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e" Oct 31 12:28:24.490776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490760 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e"} err="failed to get container status \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": rpc error: code = NotFound desc = could not find container \"b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e\": container with ID starting with b56df2749bef1e24ce5575a29156bbf576851f890414218fb0d1334d9485175e not found: ID does not exist" Oct 31 12:28:24.490818 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490775 6727 scope.go:122] "RemoveContainer" containerID="68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6" Oct 31 12:28:24.490957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490943 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6"} err="failed to get container status \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": rpc error: code = NotFound desc = could not find container \"68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6\": container with ID starting with 68cd21ecf23be2852a710701995fa2de2143b78871ab5c001829daffa3ec52b6 not found: ID does not exist" Oct 31 12:28:24.491000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.490957 6727 scope.go:122] "RemoveContainer" containerID="1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a" Oct 31 12:28:24.491147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.491129 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a"} err="failed to get container status \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": rpc error: code = NotFound desc = could not find container \"1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a\": container with ID starting with 1b8eba691102f69cb88c0c65f701a5dcb08e62b2f3f0e68e8919fdd8ba02511a not found: ID does not exist" Oct 31 12:28:24.491186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.491149 6727 scope.go:122] "RemoveContainer" containerID="014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244" Oct 31 12:28:24.491343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.491328 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244"} err="failed to get container status \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": rpc error: code = NotFound desc = could not find container \"014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244\": container with ID starting with 014b9a584cf63c5ebdaeef748f7bfdd80aafa6d269fa109c1184ab1c4a983244 not found: ID does not exist" Oct 31 12:28:24.524128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524104 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524137 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524162 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524189 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524213 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524229 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524249 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524270 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524298 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524326 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524352 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524394 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524416 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524446 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524487 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524499 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524545 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524609 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524656 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524683 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524727 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0d26075f-7f96-4d14-bf9e-85ec07189e38\" (UID: \"0d26075f-7f96-4d14-bf9e-85ec07189e38\") " Oct 31 12:28:24.524952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524861 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:24.524952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524878 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:24.524952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.524910 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:24.525336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.525314 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:24.525414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.525372 6727 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.525414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.525393 6727 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.525540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.525412 6727 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.525540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.525428 6727 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.526563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.526050 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:24.527513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.527162 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.527513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.527243 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.527684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.527568 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:28:24.527684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.527625 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:24.528288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.528251 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.528377 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.528299 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.528633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.528598 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.528727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.528665 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.528727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.528710 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:28:24.529526 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.529503 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "kube-api-access-2vx5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:28:24.529585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.529570 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:28:24.529949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.529934 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.530013 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.529991 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.538743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.538724 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config" pod "0d26075f-7f96-4d14-bf9e-85ec07189e38" (UID: "0d26075f-7f96-4d14-bf9e-85ec07189e38"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:24.626420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626373 6727 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-configmap-metrics-client-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626393 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vx5z\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-kube-api-access-2vx5z\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626403 6727 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-web-config\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626413 6727 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-db\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626421 6727 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626432 6727 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626441 6727 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-grpc-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626449 6727 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d26075f-7f96-4d14-bf9e-85ec07189e38-tls-assets\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626458 6727 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626490 6727 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626500 6727 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626508 6727 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-metrics-client-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626517 6727 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-config\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626525 6727 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d26075f-7f96-4d14-bf9e-85ec07189e38-secret-kube-rbac-proxy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626533 6727 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d26075f-7f96-4d14-bf9e-85ec07189e38-config-out\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.626627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.626541 6727 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d26075f-7f96-4d14-bf9e-85ec07189e38-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:24.755997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.755969 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:28:24.763383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.763364 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:28:24.793514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793494 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:28:24.793951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793928 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy" Oct 31 12:28:24.793951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793947 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy" Oct 31 12:28:24.793951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793955 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="prometheus" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793961 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="prometheus" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793974 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="init-config-reloader" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793980 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="init-config-reloader" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793989 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="config-reloader" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.793994 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="config-reloader" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794002 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-thanos" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794006 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-thanos" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794014 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-web" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794019 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-web" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794025 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="thanos-sidecar" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794029 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="thanos-sidecar" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794096 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794103 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="prometheus" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794109 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="init-config-reloader" Oct 31 12:28:24.794111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794115 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="config-reloader" Oct 31 12:28:24.794593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794121 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-thanos" Oct 31 12:28:24.794593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794126 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="kube-rbac-proxy-web" Oct 31 12:28:24.794593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.794132 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" containerName="thanos-sidecar" Oct 31 12:28:24.827046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.827029 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:28:24.827199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.827177 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.831695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.831675 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 12:28:24.831796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.831771 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:28:24.831924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.831902 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 12:28:24.831924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.831916 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 12:28:24.832019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.831913 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 12:28:24.832019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.831973 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 12:28:24.832304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832239 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 12:28:24.832304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832243 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 12:28:24.832304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832281 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 12:28:24.832304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832284 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 12:28:24.832304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832272 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fh91gmt3il7ar\"" Oct 31 12:28:24.832642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832315 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 12:28:24.832642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832414 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 12:28:24.832642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832511 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tmrcz\"" Oct 31 12:28:24.832781 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.832642 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 12:28:24.835948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.835924 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 12:28:24.836063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.836048 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 12:28:24.929597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929548 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929573 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929590 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929607 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929622 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929719 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929760 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49e534f-01f6-4644-9151-c7867847496d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929816 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929843 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929879 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.929921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929897 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.929958 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930028 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7p4\" (UniqueName: \"kubernetes.io/projected/e49e534f-01f6-4644-9151-c7867847496d-kube-api-access-wx7p4\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930048 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930067 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930097 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49e534f-01f6-4644-9151-c7867847496d-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930164 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930217 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-config\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930235 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:24.930295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:24.930284 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031365 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031341 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031382 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031400 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031425 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031498 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7p4\" (UniqueName: \"kubernetes.io/projected/e49e534f-01f6-4644-9151-c7867847496d-kube-api-access-wx7p4\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031528 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031550 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031638 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49e534f-01f6-4644-9151-c7867847496d-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.031908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.031887 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032026 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-config\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032068 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032126 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032166 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032185 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032189 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032223 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032254 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032279 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032302 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032328 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032345 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032360 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49e534f-01f6-4644-9151-c7867847496d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.032423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032385 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.033032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032546 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.033032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.032660 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.034331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.033892 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.035020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.034676 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49e534f-01f6-4644-9151-c7867847496d-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.035020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.034697 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.035773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.035148 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.035773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.035533 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.035773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.035592 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.035773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.035687 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.036015 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.035986 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.036098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.036049 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.036554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.036534 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-config\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.036855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.036837 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.037746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.037719 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49e534f-01f6-4644-9151-c7867847496d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.037870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.037853 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49e534f-01f6-4644-9151-c7867847496d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.038104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.038086 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49e534f-01f6-4644-9151-c7867847496d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.046511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.046486 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7p4\" (UniqueName: \"kubernetes.io/projected/e49e534f-01f6-4644-9151-c7867847496d-kube-api-access-wx7p4\") pod \"prometheus-k8s-0\" (UID: \"e49e534f-01f6-4644-9151-c7867847496d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.261069 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.261011 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:25.298139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.298114 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d26075f-7f96-4d14-bf9e-85ec07189e38" path="/var/lib/kubelet/pods/0d26075f-7f96-4d14-bf9e-85ec07189e38/volumes" Oct 31 12:28:25.578449 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:25.578368 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:28:25.582008 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:28:25.581984 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49e534f_01f6_4644_9151_c7867847496d.slice/crio-b7ee8cac19037ca349ba4bed1961d871ad82e08d727a7b777730213567579c26 WatchSource:0}: Error finding container b7ee8cac19037ca349ba4bed1961d871ad82e08d727a7b777730213567579c26: Status 404 returned error can't find the container with id b7ee8cac19037ca349ba4bed1961d871ad82e08d727a7b777730213567579c26 Oct 31 12:28:26.040450 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.040386 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:26.040450 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.040434 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:26.040614 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040522 6727 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:26.040614 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040570 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:42.04055385 +0000 UTC m=+32.855568233 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:28:26.040693 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040621 6727 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:26.040693 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040666 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:42.04065445 +0000 UTC m=+32.855668835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:28:26.040766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.040699 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:26.040812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.040775 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:26.040812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.040802 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:26.040905 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040864 6727 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:26.040905 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040868 6727 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:28:26.040905 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040871 6727 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:28:26.041018 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040911 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:42.040900517 +0000 UTC m=+32.855914900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"service-ca" not registered Oct 31 12:28:26.041018 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040932 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:42.040916331 +0000 UTC m=+32.855930723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-config" not registered Oct 31 12:28:26.041018 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.040952 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:42.040942419 +0000 UTC m=+32.855956793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:28:26.141423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.141398 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"console-57dbb8df47-wq5vh\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:26.141588 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.141571 6727 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:28:26.141639 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.141592 6727 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:28:26.141639 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.141603 6727 projected.go:196] Error preparing data for projected volume kube-api-access-ckfkv for pod openshift-console/console-57dbb8df47-wq5vh: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:26.141705 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:26.141643 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv podName:babb2014-99e4-4ae6-bf6a-ec7734ef86d0 nodeName:}" failed. No retries permitted until 2026-10-31 12:28:42.141633157 +0000 UTC m=+32.956647526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckfkv" (UniqueName: "kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv") pod "console-57dbb8df47-wq5vh" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:28:26.303189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.303140 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jq9k9" Oct 31 12:28:26.444795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.444761 6727 generic.go:356] "Generic (PLEG): container finished" podID="e49e534f-01f6-4644-9151-c7867847496d" containerID="806368d439fde419ec30359b470906ce5a52f995c9e3596a62e80e1384625b38" exitCode=0 Oct 31 12:28:26.444905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.444832 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerDied","Data":"806368d439fde419ec30359b470906ce5a52f995c9e3596a62e80e1384625b38"} Oct 31 12:28:26.444976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:26.444957 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerStarted","Data":"b7ee8cac19037ca349ba4bed1961d871ad82e08d727a7b777730213567579c26"} Oct 31 12:28:27.452659 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:27.452622 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerStarted","Data":"1b5452effa3d2914c928df6401ea466b6ae87e3367dd864dcaa9745c017c3525"} Oct 31 12:28:27.453129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:27.452685 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerStarted","Data":"5ba3ca058d879d6f0b0f5e69208e8ee9198d4ea7e396cc589655c4fd0fc16d1f"} Oct 31 12:28:27.453129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:27.452713 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerStarted","Data":"3526b61172e0f5ba3ddc353ee3e158853a72ffffe05ffcd55b1120a70864e7bb"} Oct 31 12:28:27.453129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:27.452738 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerStarted","Data":"20d67246341d9fc1392ba642bfde22e501fdc0b46c819b44bc7f75a99f8d1d87"} Oct 31 12:28:27.453129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:27.452760 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerStarted","Data":"9cfda78aa1de4993acd7f31dd448370a370b1a10250259b9a1f1ff2912b661c8"} Oct 31 12:28:27.453129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:27.452782 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49e534f-01f6-4644-9151-c7867847496d","Type":"ContainerStarted","Data":"2b249ae0cfbf1b767b26aabf083eb1ac8fe08157e54c655a8d223326044c8ae2"} Oct 31 12:28:27.488825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:27.488772 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.473721646 podStartE2EDuration="3.488758782s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.037136ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:28:27.487884903 +0000 UTC m=+18.302899295" watchObservedRunningTime="2026-10-31 12:28:27.488758782 +0000 UTC m=+18.303773173" Oct 31 12:28:30.261311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:30.261278 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:30.342156 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:28:30.342118 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:28:31.901790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:31.901753 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:32.032129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:32.032102 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:32.263197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:32.263126 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:32.390576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:32.390547 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:28:32.622131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:32.622107 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6b687" Oct 31 12:28:35.288986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.288944 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57dbb8df47-wq5vh" podUID="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" containerName="console" containerID="cri-o://be2276cab2acc8cc4b99c70d4f52723b0135292202934d79608fc36ceb092915" gracePeriod=15 Oct 31 12:28:35.487664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.487638 6727 generic.go:356] "Generic (PLEG): container finished" podID="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" containerID="be2276cab2acc8cc4b99c70d4f52723b0135292202934d79608fc36ceb092915" exitCode=0 Oct 31 12:28:35.487785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.487681 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dbb8df47-wq5vh" event={"ID":"babb2014-99e4-4ae6-bf6a-ec7734ef86d0","Type":"ContainerDied","Data":"be2276cab2acc8cc4b99c70d4f52723b0135292202934d79608fc36ceb092915"} Oct 31 12:28:35.536555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.536534 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:35.626859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.626834 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") pod \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " Oct 31 12:28:35.626986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.626882 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") pod \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " Oct 31 12:28:35.626986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.626905 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") pod \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " Oct 31 12:28:35.626986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.626971 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\") pod \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " Oct 31 12:28:35.627159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.626992 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") pod \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " Oct 31 12:28:35.627159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.627028 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") pod \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " Oct 31 12:28:35.627159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.627058 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") pod \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\" (UID: \"babb2014-99e4-4ae6-bf6a-ec7734ef86d0\") " Oct 31 12:28:35.627374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.627242 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp" pod "babb2014-99e4-4ae6-bf6a-ec7734ef86d0" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:28:35.627374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.627356 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca" pod "babb2014-99e4-4ae6-bf6a-ec7734ef86d0" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:35.627513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.627489 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config" pod "babb2014-99e4-4ae6-bf6a-ec7734ef86d0" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:35.627589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.627486 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert" pod "babb2014-99e4-4ae6-bf6a-ec7734ef86d0" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:28:35.629261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.629237 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv" pod "babb2014-99e4-4ae6-bf6a-ec7734ef86d0" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0"). InnerVolumeSpecName "kube-api-access-ckfkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:28:35.629356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.629279 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert" pod "babb2014-99e4-4ae6-bf6a-ec7734ef86d0" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:35.629397 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.629369 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config" pod "babb2014-99e4-4ae6-bf6a-ec7734ef86d0" (UID: "babb2014-99e4-4ae6-bf6a-ec7734ef86d0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:28:35.728335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.728315 6727 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-tmp\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:35.728335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.728335 6727 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-oauth-config\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:35.728455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.728345 6727 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-config\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:35.728455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.728356 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfkv\" (UniqueName: \"kubernetes.io/projected/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-kube-api-access-ckfkv\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:35.728455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.728365 6727 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-console-serving-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:35.728455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.728374 6727 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-oauth-serving-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:35.728455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:35.728382 6727 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/babb2014-99e4-4ae6-bf6a-ec7734ef86d0-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:28:36.492745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:36.492711 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dbb8df47-wq5vh" event={"ID":"babb2014-99e4-4ae6-bf6a-ec7734ef86d0","Type":"ContainerDied","Data":"5cd6628df84188ee8aee985cc8f3c34383e749b930d6526f147455d44769aec0"} Oct 31 12:28:36.492745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:36.492744 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dbb8df47-wq5vh" Oct 31 12:28:36.493201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:36.492778 6727 scope.go:122] "RemoveContainer" containerID="be2276cab2acc8cc4b99c70d4f52723b0135292202934d79608fc36ceb092915" Oct 31 12:28:36.512505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:36.512378 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57dbb8df47-wq5vh"] Oct 31 12:28:36.520696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:36.520675 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57dbb8df47-wq5vh"] Oct 31 12:28:37.302199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:37.302167 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" path="/var/lib/kubelet/pods/babb2014-99e4-4ae6-bf6a-ec7734ef86d0/volumes" Oct 31 12:28:45.376860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:45.376828 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zcknp" Oct 31 12:28:56.515126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:56.515094 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:56.530122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:56.530100 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:56.583413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:56.583389 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:28:59.906477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:28:59.906417 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:00.046761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:00.046728 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:00.340123 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:29:00.340091 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:29:03.944741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:03.944690 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:04.077809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:04.077776 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:05.885628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:05.885582 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:07.520334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:07.520301 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:08.255138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:08.255106 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:08.390967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:08.390938 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:08.872285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:08.872260 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:09.006336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:09.006303 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:16.646904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:16.646873 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:16.777944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:16.777910 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:20.561871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:20.561824 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:20.698367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:20.698297 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:22.388791 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:22.388745 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:22.523945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:29:22.523910 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:29:30.340756 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:29:30.340700 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:30:00.342197 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:30:00.342118 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:30:05.939314 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:30:05.939272 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:30:15.934171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:30:15.934132 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:30:30.339714 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:30:30.339681 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:31:00.339577 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:31:00.339537 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:31:05.999042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:05.999006 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:08.137359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:08.137315 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:23.007945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:23.007840 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:23.244538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:23.244504 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:27.374577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:27.374545 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:27.559264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:27.559227 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:30.340141 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:31:30.340110 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:31:34.768961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:34.768926 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:34.972585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:34.972550 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:36.693926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:36.693886 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:36.920424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:36.920387 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:38.391134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:38.391105 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:42.577899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:42.577859 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:42.771015 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:42.770982 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:43.783843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.783815 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b"] Oct 31 12:31:43.784280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.784261 6727 cpu_manager.go:414] "containerMap: removing container" podUID="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" containerName="console" Oct 31 12:31:43.784326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.784284 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" containerName="console" Oct 31 12:31:43.784377 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.784366 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="babb2014-99e4-4ae6-bf6a-ec7734ef86d0" containerName="console" Oct 31 12:31:43.787402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.787383 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.801998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.801971 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 12:31:43.802113 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.801996 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 12:31:43.802182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.802147 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f7jrv\"" Oct 31 12:31:43.802240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.802206 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 12:31:43.802909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.802888 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b"] Oct 31 12:31:43.844569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844544 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.844695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844574 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662xf\" (UniqueName: \"kubernetes.io/projected/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-kube-api-access-662xf\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.844695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844598 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.844695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844681 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.844873 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844824 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.844931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844878 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.844931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844908 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.845021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.844999 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.845065 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.845023 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.884322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.884298 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s"] Oct 31 12:31:43.887633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.887614 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.889641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.889616 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:43.904517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.904499 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-lqfd7\"" Oct 31 12:31:43.908644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.908624 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s"] Oct 31 12:31:43.945873 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.945852 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.945978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.945886 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.945978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.945904 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946052 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946018 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfs6x\" (UniqueName: \"kubernetes.io/projected/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-kube-api-access-pfs6x\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.946128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946108 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.946196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946180 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946216 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.946275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946242 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946245 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.946353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946296 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946336 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946441 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.946546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946512 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946562 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662xf\" (UniqueName: \"kubernetes.io/projected/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-kube-api-access-662xf\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946625 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.946699 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946654 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.946699 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946682 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.946857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946824 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.947009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946869 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:43.947009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946902 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.947009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.946979 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.947182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.947044 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.947225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.947181 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.948687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.948666 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.949108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.949093 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.957980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.957959 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:43.958199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:43.958183 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662xf\" (UniqueName: \"kubernetes.io/projected/ec9f9199-6032-4a7b-9bcf-37c42a33a41a-kube-api-access-662xf\") pod \"maas-default-gateway-openshift-default-645cf464d7-czj7b\" (UID: \"ec9f9199-6032-4a7b-9bcf-37c42a33a41a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:44.023902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.023875 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:44.048039 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.047985 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048039 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048021 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048048 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfs6x\" (UniqueName: \"kubernetes.io/projected/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-kube-api-access-pfs6x\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048074 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048187 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048216 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048261 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048297 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048322 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048343 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048578 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048607 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048645 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.048875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.048858 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.050441 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.050424 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.050700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.050678 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.060273 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.060247 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.060353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.060309 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfs6x\" (UniqueName: \"kubernetes.io/projected/eda0c4ab-b4d2-4b20-9d94-3bb0e414751c-kube-api-access-pfs6x\") pod \"openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s\" (UID: \"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.100172 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.100152 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:44.200620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.200595 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:44.375236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.375204 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b"] Oct 31 12:31:44.378770 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:31:44.378747 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9f9199_6032_4a7b_9bcf_37c42a33a41a.slice/crio-e0799066250a3fcfaa86e6180b26cfcc2a7729b2d75dde9efffeddc77e51afcd WatchSource:0}: Error finding container e0799066250a3fcfaa86e6180b26cfcc2a7729b2d75dde9efffeddc77e51afcd: Status 404 returned error can't find the container with id e0799066250a3fcfaa86e6180b26cfcc2a7729b2d75dde9efffeddc77e51afcd Oct 31 12:31:44.380664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.380644 6727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:31:44.473404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:44.473383 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s"] Oct 31 12:31:44.476000 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:31:44.475973 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda0c4ab_b4d2_4b20_9d94_3bb0e414751c.slice/crio-1f34b89ff6a576cc5bf6fcdc10aab0f9bdc1be0c0389060f707f8e3d17a584d9 WatchSource:0}: Error finding container 1f34b89ff6a576cc5bf6fcdc10aab0f9bdc1be0c0389060f707f8e3d17a584d9: Status 404 returned error can't find the container with id 1f34b89ff6a576cc5bf6fcdc10aab0f9bdc1be0c0389060f707f8e3d17a584d9 Oct 31 12:31:45.165527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:45.165493 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" event={"ID":"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c","Type":"ContainerStarted","Data":"1f34b89ff6a576cc5bf6fcdc10aab0f9bdc1be0c0389060f707f8e3d17a584d9"} Oct 31 12:31:45.166610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:45.166591 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" event={"ID":"ec9f9199-6032-4a7b-9bcf-37c42a33a41a","Type":"ContainerStarted","Data":"e0799066250a3fcfaa86e6180b26cfcc2a7729b2d75dde9efffeddc77e51afcd"} Oct 31 12:31:46.768475 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:46.768425 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:48.497614 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:48.497580 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:31:48.497999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:48.497666 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:31:48.497999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:48.497714 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:31:48.691170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:48.691132 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:31:48.691269 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:48.691207 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:31:48.691269 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:48.691236 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:31:49.184219 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:49.184187 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" event={"ID":"ec9f9199-6032-4a7b-9bcf-37c42a33a41a","Type":"ContainerStarted","Data":"c3e5c957a81e37d9b0b1139f203e79841b0656205a0d62d4fb61cc36b3bf6aa3"} Oct 31 12:31:49.185901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:49.185867 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" event={"ID":"eda0c4ab-b4d2-4b20-9d94-3bb0e414751c","Type":"ContainerStarted","Data":"32017b661d2a9b824d4cf784b4c8c1c30788697c76abc88a361204dee35b309b"} Oct 31 12:31:49.201234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:49.201205 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:49.205727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:49.205708 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:49.208522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:49.208448 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podStartSLOduration=1.8987824290000002 podStartE2EDuration="6.208434617s" totalImagesPullingTime="4.309652188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:49.208386117 +0000 UTC m=+220.023400510" watchObservedRunningTime="2026-10-31 12:31:49.208434617 +0000 UTC m=+220.023449001" Oct 31 12:31:49.230999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:49.230939 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" podStartSLOduration=2.21204484 podStartE2EDuration="6.230921522s" totalImagesPullingTime="4.018876682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:49.230364437 +0000 UTC m=+220.045378844" watchObservedRunningTime="2026-10-31 12:31:49.230921522 +0000 UTC m=+220.045935910" Oct 31 12:31:50.100391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:50.100354 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:50.104937 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:50.104911 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:50.189678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:50.189645 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:50.189875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:50.189700 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:31:50.190422 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:50.190394 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:31:50.190544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:31:50.190451 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s" Oct 31 12:32:00.341009 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:32:00.340974 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:32:06.065908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:06.065877 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:27.739029 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:27.738998 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:27.892698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:27.892661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:30.340972 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:32:30.340928 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:32:33.665709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:33.665655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:34.022612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:34.022503 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.128141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.128103 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.246438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.246407 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9"] Oct 31 12:32:35.253642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.253613 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.281889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.281861 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9"] Oct 31 12:32:35.334612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.334578 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 12:32:35.334955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.334904 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 12:32:35.335133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.335055 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b82ch\"" Oct 31 12:32:35.335133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.335116 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 12:32:35.335262 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.335220 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 12:32:35.335482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.335449 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 12:32:35.346121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.346097 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjf2g\" (UniqueName: \"kubernetes.io/projected/aa911761-d836-4067-a704-515f3158e830-kube-api-access-gjf2g\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.346224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.346133 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aa911761-d836-4067-a704-515f3158e830-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.346224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.346161 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.346307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.346268 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.346342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.346304 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.346342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.346333 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa911761-d836-4067-a704-515f3158e830-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.447194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.447110 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aa911761-d836-4067-a704-515f3158e830-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.447194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.447182 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.447435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.447314 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.447435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.447393 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.447582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.447501 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa911761-d836-4067-a704-515f3158e830-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.447582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.447565 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjf2g\" (UniqueName: \"kubernetes.io/projected/aa911761-d836-4067-a704-515f3158e830-kube-api-access-gjf2g\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.447701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.447604 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aa911761-d836-4067-a704-515f3158e830-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.449736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.449710 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa911761-d836-4067-a704-515f3158e830-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.449877 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.449855 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.450181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.450165 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.450394 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.450372 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aa911761-d836-4067-a704-515f3158e830-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.460127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.460098 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjf2g\" (UniqueName: \"kubernetes.io/projected/aa911761-d836-4067-a704-515f3158e830-kube-api-access-gjf2g\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-nj6n9\" (UID: \"aa911761-d836-4067-a704-515f3158e830\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.514905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.514856 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.568060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.568027 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:35.658615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.658584 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.795423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.795333 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.866963 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:35.866919 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9"] Oct 31 12:32:35.871323 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:32:35.871295 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa911761_d836_4067_a704_515f3158e830.slice/crio-367023060cfbde0f39845fa534d65c226e2202ccee940082f545e1f9fccf1687 WatchSource:0}: Error finding container 367023060cfbde0f39845fa534d65c226e2202ccee940082f545e1f9fccf1687: Status 404 returned error can't find the container with id 367023060cfbde0f39845fa534d65c226e2202ccee940082f545e1f9fccf1687 Oct 31 12:32:36.353101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:36.353069 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" event={"ID":"aa911761-d836-4067-a704-515f3158e830","Type":"ContainerStarted","Data":"367023060cfbde0f39845fa534d65c226e2202ccee940082f545e1f9fccf1687"} Oct 31 12:32:37.065273 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:37.065223 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:39.366740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:39.366663 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" event={"ID":"aa911761-d836-4067-a704-515f3158e830","Type":"ContainerStarted","Data":"322ef623d0e9465a5682e5f96f7ba1e8b150fb135f71d21792e8814ebf91b593"} Oct 31 12:32:40.371783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:40.371747 6727 generic.go:356] "Generic (PLEG): container finished" podID="aa911761-d836-4067-a704-515f3158e830" containerID="322ef623d0e9465a5682e5f96f7ba1e8b150fb135f71d21792e8814ebf91b593" exitCode=0 Oct 31 12:32:40.372143 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:40.371833 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" event={"ID":"aa911761-d836-4067-a704-515f3158e830","Type":"ContainerDied","Data":"322ef623d0e9465a5682e5f96f7ba1e8b150fb135f71d21792e8814ebf91b593"} Oct 31 12:32:40.503140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:40.503116 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:41.376625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:41.376592 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" event={"ID":"aa911761-d836-4067-a704-515f3158e830","Type":"ContainerStarted","Data":"b43c8b53f0929f3520f1126ac52877b9a5511e61fd962d7fa50599a5c56b3ebf"} Oct 31 12:32:41.377024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:41.376714 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:41.423283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:41.423234 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" podStartSLOduration=2.380800289 podStartE2EDuration="6.423222555s" totalImagesPullingTime="3.633158797s" totalInitContainerRuntime="409.263469ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:41.421823656 +0000 UTC m=+272.236838049" watchObservedRunningTime="2026-10-31 12:32:41.423222555 +0000 UTC m=+272.238236943" Oct 31 12:32:41.639194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:41.639126 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:42.676630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:42.676596 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:43.851658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:43.851618 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:43.990658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:43.990629 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:44.405962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:44.405914 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:45.114078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:45.114044 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:45.382250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:45.382175 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:45.953025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:45.952988 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:46.087385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:46.087352 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:47.526670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:47.526635 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:47.713208 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:47.713176 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:51.494117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:51.494039 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:52.382580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.382554 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-nj6n9" Oct 31 12:32:52.762592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.762501 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85"] Oct 31 12:32:52.767780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.767750 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:52.773209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.773161 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hnl4d\"" Oct 31 12:32:52.774171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.774146 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85"] Oct 31 12:32:52.776186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.776160 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 12:32:52.776536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.776509 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 12:32:52.886192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.886168 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6b48c32-d039-4c85-a03d-fd087c25862b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lps85\" (UID: \"a6b48c32-d039-4c85-a03d-fd087c25862b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:52.886322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.886227 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592t6\" (UniqueName: \"kubernetes.io/projected/a6b48c32-d039-4c85-a03d-fd087c25862b-kube-api-access-592t6\") pod \"cert-manager-cainjector-5c57bbd4fc-lps85\" (UID: \"a6b48c32-d039-4c85-a03d-fd087c25862b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:52.987448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.987415 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6b48c32-d039-4c85-a03d-fd087c25862b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lps85\" (UID: \"a6b48c32-d039-4c85-a03d-fd087c25862b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:52.987625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:52.987501 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592t6\" (UniqueName: \"kubernetes.io/projected/a6b48c32-d039-4c85-a03d-fd087c25862b-kube-api-access-592t6\") pod \"cert-manager-cainjector-5c57bbd4fc-lps85\" (UID: \"a6b48c32-d039-4c85-a03d-fd087c25862b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:53.004582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:53.004555 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-592t6\" (UniqueName: \"kubernetes.io/projected/a6b48c32-d039-4c85-a03d-fd087c25862b-kube-api-access-592t6\") pod \"cert-manager-cainjector-5c57bbd4fc-lps85\" (UID: \"a6b48c32-d039-4c85-a03d-fd087c25862b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:53.004884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:53.004858 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6b48c32-d039-4c85-a03d-fd087c25862b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lps85\" (UID: \"a6b48c32-d039-4c85-a03d-fd087c25862b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:53.089456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:53.089433 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" Oct 31 12:32:53.361327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:53.361263 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85"] Oct 31 12:32:53.364886 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:32:53.364861 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6b48c32_d039_4c85_a03d_fd087c25862b.slice/crio-4bdb2d1388240655dbe2abbd284e44335c84971341a1a37ada73df91091ff8d6 WatchSource:0}: Error finding container 4bdb2d1388240655dbe2abbd284e44335c84971341a1a37ada73df91091ff8d6: Status 404 returned error can't find the container with id 4bdb2d1388240655dbe2abbd284e44335c84971341a1a37ada73df91091ff8d6 Oct 31 12:32:53.426790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:53.426762 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" event={"ID":"a6b48c32-d039-4c85-a03d-fd087c25862b","Type":"ContainerStarted","Data":"4bdb2d1388240655dbe2abbd284e44335c84971341a1a37ada73df91091ff8d6"} Oct 31 12:32:54.489793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:54.489742 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:55.050940 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.050901 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:55.238530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.238496 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:55.900170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.900135 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-8drcn"] Oct 31 12:32:55.908490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.908452 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:55.911114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.911086 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2l6\" (UniqueName: \"kubernetes.io/projected/b4dec418-442c-402e-b87b-a6992ebe670c-kube-api-access-gz2l6\") pod \"cert-manager-597b95cb84-8drcn\" (UID: \"b4dec418-442c-402e-b87b-a6992ebe670c\") " pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:55.911224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.911138 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4dec418-442c-402e-b87b-a6992ebe670c-bound-sa-token\") pod \"cert-manager-597b95cb84-8drcn\" (UID: \"b4dec418-442c-402e-b87b-a6992ebe670c\") " pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:55.918951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.918561 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hzb67\"" Oct 31 12:32:55.918951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:55.918691 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8drcn"] Oct 31 12:32:56.011973 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:56.011937 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2l6\" (UniqueName: \"kubernetes.io/projected/b4dec418-442c-402e-b87b-a6992ebe670c-kube-api-access-gz2l6\") pod \"cert-manager-597b95cb84-8drcn\" (UID: \"b4dec418-442c-402e-b87b-a6992ebe670c\") " pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:56.011973 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:56.011977 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4dec418-442c-402e-b87b-a6992ebe670c-bound-sa-token\") pod \"cert-manager-597b95cb84-8drcn\" (UID: \"b4dec418-442c-402e-b87b-a6992ebe670c\") " pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:56.024513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:56.024481 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2l6\" (UniqueName: \"kubernetes.io/projected/b4dec418-442c-402e-b87b-a6992ebe670c-kube-api-access-gz2l6\") pod \"cert-manager-597b95cb84-8drcn\" (UID: \"b4dec418-442c-402e-b87b-a6992ebe670c\") " pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:56.024898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:56.024875 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4dec418-442c-402e-b87b-a6992ebe670c-bound-sa-token\") pod \"cert-manager-597b95cb84-8drcn\" (UID: \"b4dec418-442c-402e-b87b-a6992ebe670c\") " pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:56.224412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:56.224344 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8drcn" Oct 31 12:32:56.510395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:56.510311 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8drcn"] Oct 31 12:32:56.513545 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:32:56.513513 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4dec418_442c_402e_b87b_a6992ebe670c.slice/crio-93da33326aaf91a9fb08c455eb3f8be0a3b0c15e32925a986a489a59644de81f WatchSource:0}: Error finding container 93da33326aaf91a9fb08c455eb3f8be0a3b0c15e32925a986a489a59644de81f: Status 404 returned error can't find the container with id 93da33326aaf91a9fb08c455eb3f8be0a3b0c15e32925a986a489a59644de81f Oct 31 12:32:57.456652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:57.456612 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8drcn" event={"ID":"b4dec418-442c-402e-b87b-a6992ebe670c","Type":"ContainerStarted","Data":"93da33326aaf91a9fb08c455eb3f8be0a3b0c15e32925a986a489a59644de81f"} Oct 31 12:32:58.463147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:58.463120 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" event={"ID":"a6b48c32-d039-4c85-a03d-fd087c25862b","Type":"ContainerStarted","Data":"99611a1f07df45b28dc5f0d97a318271ef35d313496e15cb2733cd1b3ce726be"} Oct 31 12:32:58.482792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:58.482746 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lps85" podStartSLOduration=2.407198334 podStartE2EDuration="6.482734775s" totalImagesPullingTime="4.075536441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:58.482488802 +0000 UTC m=+289.297503194" watchObservedRunningTime="2026-10-31 12:32:58.482734775 +0000 UTC m=+289.297749156" Oct 31 12:32:58.638021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:58.637992 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:58.792995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:58.792921 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:59.364430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.364400 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4"] Oct 31 12:32:59.367758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.367737 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.376112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.376091 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nszgg\"" Oct 31 12:32:59.376326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.376301 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 12:32:59.376404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.376355 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 12:32:59.376448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.376412 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 12:32:59.376448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.376417 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 12:32:59.379111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.379089 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4"] Oct 31 12:32:59.412206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.412176 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:59.441034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.441010 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnzh\" (UniqueName: \"kubernetes.io/projected/495629b0-eb86-4e37-9941-363270905cd1-kube-api-access-rvnzh\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.441131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.441053 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/495629b0-eb86-4e37-9941-363270905cd1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.441131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.441074 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/495629b0-eb86-4e37-9941-363270905cd1-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.473804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.473770 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8drcn" event={"ID":"b4dec418-442c-402e-b87b-a6992ebe670c","Type":"ContainerStarted","Data":"990971ebd5813e781f7fc70da8627819a748950a57f386e987c1fbbdb333b304"} Oct 31 12:32:59.494503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.494439 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-8drcn" podStartSLOduration=2.577420964 podStartE2EDuration="4.494423273s" totalImagesPullingTime="1.917002309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:59.493640435 +0000 UTC m=+290.308654827" watchObservedRunningTime="2026-10-31 12:32:59.494423273 +0000 UTC m=+290.309437664" Oct 31 12:32:59.541848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.541813 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnzh\" (UniqueName: \"kubernetes.io/projected/495629b0-eb86-4e37-9941-363270905cd1-kube-api-access-rvnzh\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.542009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.541867 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/495629b0-eb86-4e37-9941-363270905cd1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.542009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.541899 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/495629b0-eb86-4e37-9941-363270905cd1-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.542731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.542706 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/495629b0-eb86-4e37-9941-363270905cd1-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.544275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.544251 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/495629b0-eb86-4e37-9941-363270905cd1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.549228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.549207 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:59.555991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.555969 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnzh\" (UniqueName: \"kubernetes.io/projected/495629b0-eb86-4e37-9941-363270905cd1-kube-api-access-rvnzh\") pod \"kuadrant-console-plugin-66c4fb575d-r5qv4\" (UID: \"495629b0-eb86-4e37-9941-363270905cd1\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.679341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.679246 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" Oct 31 12:32:59.954809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:32:59.954737 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4"] Oct 31 12:32:59.959082 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:32:59.959054 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod495629b0_eb86_4e37_9941_363270905cd1.slice/crio-584878163f93785bb8306726619c5df77b899001e23ac8af7f51e67ad62c2e97 WatchSource:0}: Error finding container 584878163f93785bb8306726619c5df77b899001e23ac8af7f51e67ad62c2e97: Status 404 returned error can't find the container with id 584878163f93785bb8306726619c5df77b899001e23ac8af7f51e67ad62c2e97 Oct 31 12:33:00.341925 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:33:00.341884 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:33:00.479728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:00.479694 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" event={"ID":"495629b0-eb86-4e37-9941-363270905cd1","Type":"ContainerStarted","Data":"584878163f93785bb8306726619c5df77b899001e23ac8af7f51e67ad62c2e97"} Oct 31 12:33:05.504286 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:05.504245 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" event={"ID":"495629b0-eb86-4e37-9941-363270905cd1","Type":"ContainerStarted","Data":"b616373ae2c0688c756c9e58cba9d288310ef3d02adab9a2e93ee1ed3e36d479"} Oct 31 12:33:05.522863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:05.522820 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-r5qv4" podStartSLOduration=1.827350969 podStartE2EDuration="6.522805835s" totalImagesPullingTime="4.695454866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:05.522125612 +0000 UTC m=+296.337140003" watchObservedRunningTime="2026-10-31 12:33:05.522805835 +0000 UTC m=+296.337820220" Oct 31 12:33:05.694668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:05.694638 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:06.117639 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:06.117603 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:06.248058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:06.248015 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:06.946197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:06.946159 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:07.095285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.095259 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:07.387385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.387356 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9"] Oct 31 12:33:07.391829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.391812 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.406159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.406130 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9"] Oct 31 12:33:07.409405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.409380 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcnpg\" (UniqueName: \"kubernetes.io/projected/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-kube-api-access-wcnpg\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.409548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.409442 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-tls-certs\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.409617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.409582 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-tmp\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.410892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.410874 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 12:33:07.411221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.411206 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 12:33:07.411290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.411276 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qprtw\"" Oct 31 12:33:07.421314 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.421248 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:07.510666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.510643 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-tmp\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.510776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.510691 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcnpg\" (UniqueName: \"kubernetes.io/projected/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-kube-api-access-wcnpg\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.510776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.510727 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-tls-certs\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.512735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.512704 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-tmp\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.513011 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.512995 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-tls-certs\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.525851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.525821 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcnpg\" (UniqueName: \"kubernetes.io/projected/aa2c139d-e9d9-42c2-a78f-c741002c0dd2-kube-api-access-wcnpg\") pod \"kube-auth-proxy-57cf7fb5f5-rr2g9\" (UID: \"aa2c139d-e9d9-42c2-a78f-c741002c0dd2\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.550774 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.550751 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv"] Oct 31 12:33:07.554216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.554199 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.563911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.563891 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bcpwr\"" Oct 31 12:33:07.577686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.577666 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv"] Oct 31 12:33:07.611683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.611661 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.611782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.611701 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-data\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.611782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.611734 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.611878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.611783 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4af62-beef-47b2-85b6-ee18be5cac62-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.611923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.611894 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.611971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.611929 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-token\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.611971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.611959 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.612062 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.612042 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgppg\" (UniqueName: \"kubernetes.io/projected/2fc4af62-beef-47b2-85b6-ee18be5cac62-kube-api-access-fgppg\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.612104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.612067 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.623892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.623869 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:07.703099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.703046 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" Oct 31 12:33:07.713019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.712999 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgppg\" (UniqueName: \"kubernetes.io/projected/2fc4af62-beef-47b2-85b6-ee18be5cac62-kube-api-access-fgppg\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713035 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713060 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713109 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-data\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713159 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713181 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4af62-beef-47b2-85b6-ee18be5cac62-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713217 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713250 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-token\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.713332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713289 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.714270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713546 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.714270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713597 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.714270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.713661 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.714270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.714111 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4af62-beef-47b2-85b6-ee18be5cac62-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.714270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.714273 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-data\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.715789 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.715770 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.715879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.715769 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.724861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.724842 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4af62-beef-47b2-85b6-ee18be5cac62-istio-token\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.725138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.725119 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgppg\" (UniqueName: \"kubernetes.io/projected/2fc4af62-beef-47b2-85b6-ee18be5cac62-kube-api-access-fgppg\") pod \"data-science-gateway-data-science-gateway-class-848cd468bfrdfjv\" (UID: \"2fc4af62-beef-47b2-85b6-ee18be5cac62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.775496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.773681 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:07.867254 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.867220 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:07.976633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:07.976566 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9"] Oct 31 12:33:07.979427 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:33:07.979402 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2c139d_e9d9_42c2_a78f_c741002c0dd2.slice/crio-b38890352b2b0e9e4d74b750c309fba37afedd56fa2a3894d2ff1e507097adb6 WatchSource:0}: Error finding container b38890352b2b0e9e4d74b750c309fba37afedd56fa2a3894d2ff1e507097adb6: Status 404 returned error can't find the container with id b38890352b2b0e9e4d74b750c309fba37afedd56fa2a3894d2ff1e507097adb6 Oct 31 12:33:08.157255 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.157231 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv"] Oct 31 12:33:08.159939 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:33:08.159903 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc4af62_beef_47b2_85b6_ee18be5cac62.slice/crio-72a336a15a52777c0164097207dd3d72110c075b59484975fa92c4b18d8385cc WatchSource:0}: Error finding container 72a336a15a52777c0164097207dd3d72110c075b59484975fa92c4b18d8385cc: Status 404 returned error can't find the container with id 72a336a15a52777c0164097207dd3d72110c075b59484975fa92c4b18d8385cc Oct 31 12:33:08.162055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.162021 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:33:08.162136 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.162086 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:33:08.162136 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.162115 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:33:08.277510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.277429 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:08.448209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.448174 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:08.517631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.517602 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" event={"ID":"2fc4af62-beef-47b2-85b6-ee18be5cac62","Type":"ContainerStarted","Data":"01e5af6fc14ee73b3907fd8c6d6b48942be1711b2e6d5c41c4a8d611cf857ee0"} Oct 31 12:33:08.517806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.517660 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" event={"ID":"2fc4af62-beef-47b2-85b6-ee18be5cac62","Type":"ContainerStarted","Data":"72a336a15a52777c0164097207dd3d72110c075b59484975fa92c4b18d8385cc"} Oct 31 12:33:08.518884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.518860 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" event={"ID":"aa2c139d-e9d9-42c2-a78f-c741002c0dd2","Type":"ContainerStarted","Data":"b38890352b2b0e9e4d74b750c309fba37afedd56fa2a3894d2ff1e507097adb6"} Oct 31 12:33:08.543056 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.542978 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" podStartSLOduration=1.542965938 podStartE2EDuration="1.542965938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:08.541559739 +0000 UTC m=+299.356574129" watchObservedRunningTime="2026-10-31 12:33:08.542965938 +0000 UTC m=+299.357980326" Oct 31 12:33:08.867863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.867832 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:08.924910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:08.924849 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:09.299622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:09.299547 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:09.551811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:09.551718 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:09.745655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:09.745561 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:09.876659 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:09.876622 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:10.527821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:10.527787 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:10.528975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:10.528951 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-848cd468bfrdfjv" Oct 31 12:33:10.645268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:10.645241 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:10.691124 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:10.691102 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 12:33:10.815824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:10.815762 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:11.533820 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.533780 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" event={"ID":"aa2c139d-e9d9-42c2-a78f-c741002c0dd2","Type":"ContainerStarted","Data":"4e54bcf6d8bce0024798095ebcedbb7930f19e6afc919e6a43410c233809fddb"} Oct 31 12:33:11.554478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.554416 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-rr2g9" podStartSLOduration=1.8499497900000001 podStartE2EDuration="4.554401503s" totalImagesPullingTime="2.704451713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:11.554223497 +0000 UTC m=+302.369237889" watchObservedRunningTime="2026-10-31 12:33:11.554401503 +0000 UTC m=+302.369415897" Oct 31 12:33:11.820408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.820381 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6"] Oct 31 12:33:11.823815 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.823799 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.845897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.845877 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 12:33:11.846185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.846170 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 12:33:11.846440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.846429 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-t9qhb\"" Oct 31 12:33:11.849494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.849479 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 12:33:11.856528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.856505 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d96d359f-1732-4408-b38e-e2d0d9143950-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.856729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.856700 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96d359f-1732-4408-b38e-e2d0d9143950-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.856824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.856788 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d96d359f-1732-4408-b38e-e2d0d9143950-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.856897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.856868 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpgf\" (UniqueName: \"kubernetes.io/projected/d96d359f-1732-4408-b38e-e2d0d9143950-kube-api-access-bqpgf\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.857755 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.857732 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6"] Oct 31 12:33:11.957493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.957449 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d96d359f-1732-4408-b38e-e2d0d9143950-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.957618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.957518 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96d359f-1732-4408-b38e-e2d0d9143950-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.957696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.957670 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d96d359f-1732-4408-b38e-e2d0d9143950-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.957751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.957709 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpgf\" (UniqueName: \"kubernetes.io/projected/d96d359f-1732-4408-b38e-e2d0d9143950-kube-api-access-bqpgf\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.957854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.957836 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96d359f-1732-4408-b38e-e2d0d9143950-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.958000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.957980 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d96d359f-1732-4408-b38e-e2d0d9143950-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.958450 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.958427 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d96d359f-1732-4408-b38e-e2d0d9143950-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:11.983036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:11.983015 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpgf\" (UniqueName: \"kubernetes.io/projected/d96d359f-1732-4408-b38e-e2d0d9143950-kube-api-access-bqpgf\") pod \"kserve-module-controller-manager-7566cf5fc8-sbzv6\" (UID: \"d96d359f-1732-4408-b38e-e2d0d9143950\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:12.136641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:12.136583 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:12.417284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:12.417205 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6"] Oct 31 12:33:12.421040 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:33:12.421004 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96d359f_1732_4408_b38e_e2d0d9143950.slice/crio-163e1b23f6734163d0c7b39d462fb30e98c24e30ad73189cb0d3ef3e25345e95 WatchSource:0}: Error finding container 163e1b23f6734163d0c7b39d462fb30e98c24e30ad73189cb0d3ef3e25345e95: Status 404 returned error can't find the container with id 163e1b23f6734163d0c7b39d462fb30e98c24e30ad73189cb0d3ef3e25345e95 Oct 31 12:33:12.538758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:12.538728 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" event={"ID":"d96d359f-1732-4408-b38e-e2d0d9143950","Type":"ContainerStarted","Data":"163e1b23f6734163d0c7b39d462fb30e98c24e30ad73189cb0d3ef3e25345e95"} Oct 31 12:33:13.326779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:13.326751 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:13.534537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:13.534505 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:14.548444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:14.548409 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" event={"ID":"d96d359f-1732-4408-b38e-e2d0d9143950","Type":"ContainerStarted","Data":"fd0ddc1adecbf2955b7d3210e876850681baeed0364696fc19f2a06d062f4dcb"} Oct 31 12:33:14.548807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:14.548523 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:14.571701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:14.571654 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" podStartSLOduration=1.635977257 podStartE2EDuration="3.571641816s" totalImagesPullingTime="1.935664559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:14.571085253 +0000 UTC m=+305.386099640" watchObservedRunningTime="2026-10-31 12:33:14.571641816 +0000 UTC m=+305.386656207" Oct 31 12:33:19.080363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:19.080325 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:19.235284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:19.235250 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:19.371341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:19.371255 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:22.601173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:22.601141 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:22.799088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:22.799055 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:27.411214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:27.411179 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:27.581788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:27.581758 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:29.803431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:29.803403 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:29.939686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:29.939651 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:30.340348 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:33:30.340316 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:33:31.929177 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:31.929146 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:32.119334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:32.119291 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:33.940044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:33.940011 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:34.152090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:34.152049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:35.555140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:35.555109 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbzv6" Oct 31 12:33:37.201496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.201444 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:37.509445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.509370 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:33:37.531290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.531264 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:33:37.531429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.531350 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:37.547733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.547710 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:33:37.594897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.594869 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 12:33:37.608943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.608915 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8sj\" (UniqueName: \"kubernetes.io/projected/14f5ae26-97dd-4477-9128-060c0a975982-kube-api-access-hd8sj\") pod \"limitador-limitador-7c9b87d574-cbmjj\" (UID: \"14f5ae26-97dd-4477-9128-060c0a975982\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:37.609015 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.608974 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/14f5ae26-97dd-4477-9128-060c0a975982-config-file\") pod \"limitador-limitador-7c9b87d574-cbmjj\" (UID: \"14f5ae26-97dd-4477-9128-060c0a975982\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:37.709657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.709625 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8sj\" (UniqueName: \"kubernetes.io/projected/14f5ae26-97dd-4477-9128-060c0a975982-kube-api-access-hd8sj\") pod \"limitador-limitador-7c9b87d574-cbmjj\" (UID: \"14f5ae26-97dd-4477-9128-060c0a975982\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:37.709819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.709689 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/14f5ae26-97dd-4477-9128-060c0a975982-config-file\") pod \"limitador-limitador-7c9b87d574-cbmjj\" (UID: \"14f5ae26-97dd-4477-9128-060c0a975982\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:37.710301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.710276 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/14f5ae26-97dd-4477-9128-060c0a975982-config-file\") pod \"limitador-limitador-7c9b87d574-cbmjj\" (UID: \"14f5ae26-97dd-4477-9128-060c0a975982\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:37.722777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.722751 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8sj\" (UniqueName: \"kubernetes.io/projected/14f5ae26-97dd-4477-9128-060c0a975982-kube-api-access-hd8sj\") pod \"limitador-limitador-7c9b87d574-cbmjj\" (UID: \"14f5ae26-97dd-4477-9128-060c0a975982\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:37.845657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:37.845624 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:38.118179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:38.118108 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:33:38.121628 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:33:38.121595 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f5ae26_97dd_4477_9128_060c0a975982.slice/crio-5243053b14926946d36f12162f5cddff180952230d9f3bff7c610f1f8140025c WatchSource:0}: Error finding container 5243053b14926946d36f12162f5cddff180952230d9f3bff7c610f1f8140025c: Status 404 returned error can't find the container with id 5243053b14926946d36f12162f5cddff180952230d9f3bff7c610f1f8140025c Oct 31 12:33:38.656631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:38.656595 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" event={"ID":"14f5ae26-97dd-4477-9128-060c0a975982","Type":"ContainerStarted","Data":"5243053b14926946d36f12162f5cddff180952230d9f3bff7c610f1f8140025c"} Oct 31 12:33:40.409218 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:40.409182 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:40.569407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:40.569380 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:40.667135 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:40.667047 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" event={"ID":"14f5ae26-97dd-4477-9128-060c0a975982","Type":"ContainerStarted","Data":"caf6feec4466482326be837ceb2a89b1c5adb669f18284019d4efc1e228c46a4"} Oct 31 12:33:40.667279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:40.667195 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:40.688418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:40.688375 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" podStartSLOduration=1.373246921 podStartE2EDuration="3.688361529s" totalImagesPullingTime="2.315114608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:40.687576798 +0000 UTC m=+331.502591190" watchObservedRunningTime="2026-10-31 12:33:40.688361529 +0000 UTC m=+331.503375980" Oct 31 12:33:43.422960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:43.422922 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:43.562253 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:43.562215 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:44.710938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:44.710907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:44.851160 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:44.851127 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:48.861968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:48.861915 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:49.060711 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:49.060675 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:49.229047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:49.228972 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:49.372455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:49.372411 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:51.672504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:51.672472 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-cbmjj" Oct 31 12:33:55.413679 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:55.413641 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:55.565191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:55.565149 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:56.159305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:56.159264 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:56.609613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:56.609584 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:56.858481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:33:56.858418 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:00.342776 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:34:00.342742 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:34:02.068102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:02.068071 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:02.318216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:02.318113 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:05.429893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:05.429823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:05.903390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:05.903361 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:06.152355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.152322 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:06.327036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.327004 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6zm6w"] Oct 31 12:34:06.333869 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.333847 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.356305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.356274 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6zm6w"] Oct 31 12:34:06.402240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.402214 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 12:34:06.402240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.402216 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4jlkl\"" Oct 31 12:34:06.425198 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.425168 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:06.460342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.460318 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ggd\" (UniqueName: \"kubernetes.io/projected/490fd3c3-446e-40cf-a5ac-ebf98509a290-kube-api-access-s2ggd\") pod \"model-serving-api-75bd89fdfc-6zm6w\" (UID: \"490fd3c3-446e-40cf-a5ac-ebf98509a290\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.460679 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.460378 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490fd3c3-446e-40cf-a5ac-ebf98509a290-tls-certs\") pod \"model-serving-api-75bd89fdfc-6zm6w\" (UID: \"490fd3c3-446e-40cf-a5ac-ebf98509a290\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.561741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.561721 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490fd3c3-446e-40cf-a5ac-ebf98509a290-tls-certs\") pod \"model-serving-api-75bd89fdfc-6zm6w\" (UID: \"490fd3c3-446e-40cf-a5ac-ebf98509a290\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.561829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.561774 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ggd\" (UniqueName: \"kubernetes.io/projected/490fd3c3-446e-40cf-a5ac-ebf98509a290-kube-api-access-s2ggd\") pod \"model-serving-api-75bd89fdfc-6zm6w\" (UID: \"490fd3c3-446e-40cf-a5ac-ebf98509a290\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.564118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.564092 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490fd3c3-446e-40cf-a5ac-ebf98509a290-tls-certs\") pod \"model-serving-api-75bd89fdfc-6zm6w\" (UID: \"490fd3c3-446e-40cf-a5ac-ebf98509a290\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.581618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.581563 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ggd\" (UniqueName: \"kubernetes.io/projected/490fd3c3-446e-40cf-a5ac-ebf98509a290-kube-api-access-s2ggd\") pod \"model-serving-api-75bd89fdfc-6zm6w\" (UID: \"490fd3c3-446e-40cf-a5ac-ebf98509a290\") " pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.650619 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.650594 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:06.722059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.721956 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:06.932087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.932011 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6zm6w"] Oct 31 12:34:06.936528 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:34:06.936478 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490fd3c3_446e_40cf_a5ac_ebf98509a290.slice/crio-66972c97389f321208bf37999576d10d50bf9880e2cf2449206ac0c5e5d4accc WatchSource:0}: Error finding container 66972c97389f321208bf37999576d10d50bf9880e2cf2449206ac0c5e5d4accc: Status 404 returned error can't find the container with id 66972c97389f321208bf37999576d10d50bf9880e2cf2449206ac0c5e5d4accc Oct 31 12:34:06.999037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:06.999015 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:07.789107 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:07.789062 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" event={"ID":"490fd3c3-446e-40cf-a5ac-ebf98509a290","Type":"ContainerStarted","Data":"66972c97389f321208bf37999576d10d50bf9880e2cf2449206ac0c5e5d4accc"} Oct 31 12:34:08.082028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:08.081994 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:08.292055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:08.292028 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:08.797344 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:08.797314 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" event={"ID":"490fd3c3-446e-40cf-a5ac-ebf98509a290","Type":"ContainerStarted","Data":"af6e0799332189a0898fee5e0833ab444df2309b26545a2c175fbdf0b9594d67"} Oct 31 12:34:08.797666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:08.797400 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:08.842596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:08.842559 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" podStartSLOduration=1.120380896 podStartE2EDuration="2.842546036s" totalImagesPullingTime="1.72216514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:08.841446855 +0000 UTC m=+359.656461246" watchObservedRunningTime="2026-10-31 12:34:08.842546036 +0000 UTC m=+359.657560427" Oct 31 12:34:15.954661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:15.954611 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:16.021748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.021708 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-lhz9k"] Oct 31 12:34:16.031322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.031294 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-lhz9k"] Oct 31 12:34:16.031478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.031420 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.038443 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.038419 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 12:34:16.038631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.038615 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-t8k9s\"" Oct 31 12:34:16.039648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.039631 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 12:34:16.039727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.039679 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 12:34:16.115418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.115398 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:16.151190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.151166 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-webhook-cert\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.151277 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.151211 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-metrics-certs\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.151277 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.151233 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kph\" (UniqueName: \"kubernetes.io/projected/3f77646d-be36-45b8-bfdc-8e44e17b91bf-kube-api-access-77kph\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.241627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.241573 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:16.251788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.251771 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-metrics-certs\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.251857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.251801 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kph\" (UniqueName: \"kubernetes.io/projected/3f77646d-be36-45b8-bfdc-8e44e17b91bf-kube-api-access-77kph\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.251894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.251853 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-webhook-cert\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.254249 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.254226 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-metrics-certs\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.254298 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.254255 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-webhook-cert\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.265108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.265084 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kph\" (UniqueName: \"kubernetes.io/projected/3f77646d-be36-45b8-bfdc-8e44e17b91bf-kube-api-access-77kph\") pod \"maas-controller-579cf7c6d-lhz9k\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.337647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.337623 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64d9577675-zr4jx"] Oct 31 12:34:16.341235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.341220 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.344548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.344524 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:16.354583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.354555 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d9577675-zr4jx"] Oct 31 12:34:16.368609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.368579 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmj2\" (UniqueName: \"kubernetes.io/projected/5c4ced04-45b1-440a-87c2-e0d5f595981a-kube-api-access-nrmj2\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.368731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.368641 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-webhook-cert\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.368731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.368711 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-metrics-certs\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.469181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.469146 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmj2\" (UniqueName: \"kubernetes.io/projected/5c4ced04-45b1-440a-87c2-e0d5f595981a-kube-api-access-nrmj2\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.469336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.469196 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-webhook-cert\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.469336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.469232 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-metrics-certs\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.471666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.471641 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-metrics-certs\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.471796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.471776 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-webhook-cert\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.483542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.483520 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmj2\" (UniqueName: \"kubernetes.io/projected/5c4ced04-45b1-440a-87c2-e0d5f595981a-kube-api-access-nrmj2\") pod \"maas-controller-64d9577675-zr4jx\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.632515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.632491 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-lhz9k"] Oct 31 12:34:16.635160 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:34:16.635135 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f77646d_be36_45b8_bfdc_8e44e17b91bf.slice/crio-f5cc3baf38ac68b939b90127c154523d13a9aaaa8094c4e52f84dc6a6d43e283 WatchSource:0}: Error finding container f5cc3baf38ac68b939b90127c154523d13a9aaaa8094c4e52f84dc6a6d43e283: Status 404 returned error can't find the container with id f5cc3baf38ac68b939b90127c154523d13a9aaaa8094c4e52f84dc6a6d43e283 Oct 31 12:34:16.659247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.659226 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:16.835156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.835121 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" event={"ID":"3f77646d-be36-45b8-bfdc-8e44e17b91bf","Type":"ContainerStarted","Data":"f5cc3baf38ac68b939b90127c154523d13a9aaaa8094c4e52f84dc6a6d43e283"} Oct 31 12:34:16.932976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:16.932952 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d9577675-zr4jx"] Oct 31 12:34:16.935517 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:34:16.935493 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4ced04_45b1_440a_87c2_e0d5f595981a.slice/crio-93b9a43d55882416e8b8a60566bd5fc3f12b8aa75b92581d05153e1d96e8669e WatchSource:0}: Error finding container 93b9a43d55882416e8b8a60566bd5fc3f12b8aa75b92581d05153e1d96e8669e: Status 404 returned error can't find the container with id 93b9a43d55882416e8b8a60566bd5fc3f12b8aa75b92581d05153e1d96e8669e Oct 31 12:34:17.066805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:17.066777 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:17.847452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:17.847412 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-zr4jx" event={"ID":"5c4ced04-45b1-440a-87c2-e0d5f595981a","Type":"ContainerStarted","Data":"93b9a43d55882416e8b8a60566bd5fc3f12b8aa75b92581d05153e1d96e8669e"} Oct 31 12:34:19.806487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:19.805851 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6zm6w" Oct 31 12:34:20.434007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.433899 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:20.611326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.610589 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:20.864112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.864073 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" event={"ID":"3f77646d-be36-45b8-bfdc-8e44e17b91bf","Type":"ContainerStarted","Data":"9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb"} Oct 31 12:34:20.864598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.864189 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:20.865770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.865741 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-zr4jx" event={"ID":"5c4ced04-45b1-440a-87c2-e0d5f595981a","Type":"ContainerStarted","Data":"f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd"} Oct 31 12:34:20.865906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.865844 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:20.895683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.895628 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" podStartSLOduration=2.388913553 podStartE2EDuration="5.895615524s" totalImagesPullingTime="3.506701971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:20.894332557 +0000 UTC m=+371.709346962" watchObservedRunningTime="2026-10-31 12:34:20.895615524 +0000 UTC m=+371.710629915" Oct 31 12:34:20.923560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:20.923513 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64d9577675-zr4jx" podStartSLOduration=1.678550304 podStartE2EDuration="4.923500188s" totalImagesPullingTime="3.244949884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:20.923028811 +0000 UTC m=+371.738043202" watchObservedRunningTime="2026-10-31 12:34:20.923500188 +0000 UTC m=+371.738514582" Oct 31 12:34:22.799214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:22.799180 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:22.937065 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:22.937029 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:30.341199 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:34:30.341165 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:34:31.872814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:31.872785 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:31.873245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:31.872858 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 12:34:31.967048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:31.967016 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-lhz9k"] Oct 31 12:34:31.967258 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:31.967231 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" podUID="3f77646d-be36-45b8-bfdc-8e44e17b91bf" containerName="manager" containerID="cri-o://9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb" gracePeriod=30 Oct 31 12:34:32.208136 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.208111 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:32.307839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.307813 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-metrics-certs\") pod \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " Oct 31 12:34:32.307987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.307858 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-webhook-cert\") pod \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " Oct 31 12:34:32.307987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.307887 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f77646d-be36-45b8-bfdc-8e44e17b91bf-kube-api-access-77kph\" (UniqueName: \"kubernetes.io/projected/3f77646d-be36-45b8-bfdc-8e44e17b91bf-kube-api-access-77kph\") pod \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\" (UID: \"3f77646d-be36-45b8-bfdc-8e44e17b91bf\") " Oct 31 12:34:32.309965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.309938 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-metrics-certs" pod "3f77646d-be36-45b8-bfdc-8e44e17b91bf" (UID: "3f77646d-be36-45b8-bfdc-8e44e17b91bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:34:32.310077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.309965 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-webhook-cert" pod "3f77646d-be36-45b8-bfdc-8e44e17b91bf" (UID: "3f77646d-be36-45b8-bfdc-8e44e17b91bf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:34:32.310077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.310005 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f77646d-be36-45b8-bfdc-8e44e17b91bf-kube-api-access-77kph" pod "3f77646d-be36-45b8-bfdc-8e44e17b91bf" (UID: "3f77646d-be36-45b8-bfdc-8e44e17b91bf"). InnerVolumeSpecName "kube-api-access-77kph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:34:32.409154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.409097 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:32.409154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.409119 6727 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f77646d-be36-45b8-bfdc-8e44e17b91bf-webhook-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:32.409154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.409129 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kph\" (UniqueName: \"kubernetes.io/projected/3f77646d-be36-45b8-bfdc-8e44e17b91bf-kube-api-access-77kph\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:32.921439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.921408 6727 generic.go:356] "Generic (PLEG): container finished" podID="3f77646d-be36-45b8-bfdc-8e44e17b91bf" containerID="9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb" exitCode=0 Oct 31 12:34:32.921824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.921452 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" Oct 31 12:34:32.921824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.921501 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" event={"ID":"3f77646d-be36-45b8-bfdc-8e44e17b91bf","Type":"ContainerDied","Data":"9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb"} Oct 31 12:34:32.921824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.921553 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-lhz9k" event={"ID":"3f77646d-be36-45b8-bfdc-8e44e17b91bf","Type":"ContainerDied","Data":"f5cc3baf38ac68b939b90127c154523d13a9aaaa8094c4e52f84dc6a6d43e283"} Oct 31 12:34:32.921971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.921777 6727 scope.go:122] "RemoveContainer" containerID="9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb" Oct 31 12:34:32.936339 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.936312 6727 scope.go:122] "RemoveContainer" containerID="9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb" Oct 31 12:34:32.936618 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:34:32.936598 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb\": container with ID starting with 9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb not found: ID does not exist" containerID="9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb" Oct 31 12:34:32.936710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.936656 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb"} err="failed to get container status \"9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb\": rpc error: code = NotFound desc = could not find container \"9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb\": container with ID starting with 9faa0238e342f4327826680c2fc062022051f08fe43da07c30a81cbaa2374eeb not found: ID does not exist" Oct 31 12:34:32.949723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.949701 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-lhz9k"] Oct 31 12:34:32.972823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:32.972803 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-lhz9k"] Oct 31 12:34:33.298327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:33.298271 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f77646d-be36-45b8-bfdc-8e44e17b91bf" path="/var/lib/kubelet/pods/3f77646d-be36-45b8-bfdc-8e44e17b91bf/volumes" Oct 31 12:34:50.856012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:50.855975 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:51.035202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:51.035171 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:52.147016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:52.146974 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:52.313875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:52.313840 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:52.863389 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:34:52.863356 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:35:00.340977 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:35:00.340930 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:35:01.068607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:01.068567 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:02.052222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.052150 6727 generic.go:356] "Generic (PLEG): container finished" podID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerID="c3e5c957a81e37d9b0b1139f203e79841b0656205a0d62d4fb61cc36b3bf6aa3" exitCode=0 Oct 31 12:35:02.052597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.052225 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" event={"ID":"ec9f9199-6032-4a7b-9bcf-37c42a33a41a","Type":"ContainerDied","Data":"c3e5c957a81e37d9b0b1139f203e79841b0656205a0d62d4fb61cc36b3bf6aa3"} Oct 31 12:35:02.052597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.052564 6727 scope.go:122] "RemoveContainer" containerID="c3e5c957a81e37d9b0b1139f203e79841b0656205a0d62d4fb61cc36b3bf6aa3" Oct 31 12:35:02.053637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.053606 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:35:02.053704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.053678 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:35:02.053745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.053709 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:35:02.643187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.643149 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:02.789024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:02.788984 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:03.059046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.058964 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" event={"ID":"ec9f9199-6032-4a7b-9bcf-37c42a33a41a","Type":"ContainerStarted","Data":"5e108cb6b307d721e7c83e589bf67e1ad8cc92dbcbd4a604fddf89f1b655cc71"} Oct 31 12:35:03.101412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.101381 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:35:03.103265 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.103237 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body= Oct 31 12:35:03.103425 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.103299 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" Oct 31 12:35:03.125387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.125361 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-kstmg"] Oct 31 12:35:03.125754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.125742 6727 cpu_manager.go:414] "containerMap: removing container" podUID="3f77646d-be36-45b8-bfdc-8e44e17b91bf" containerName="manager" Oct 31 12:35:03.125800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.125756 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f77646d-be36-45b8-bfdc-8e44e17b91bf" containerName="manager" Oct 31 12:35:03.125858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.125848 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f77646d-be36-45b8-bfdc-8e44e17b91bf" containerName="manager" Oct 31 12:35:03.128364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.128343 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.132273 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.132254 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\"" Oct 31 12:35:03.132654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.132636 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 12:35:03.134316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.134295 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 12:35:03.134574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.134557 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:35:03.134709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.134695 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:35:03.147868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.147849 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-kstmg"] Oct 31 12:35:03.191404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.191361 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2bs\" (UniqueName: \"kubernetes.io/projected/8cc1277f-a661-4d8c-8402-4b90554f0ba9-kube-api-access-fl2bs\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.191603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.191498 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-metrics-certs\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.191681 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.191641 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-maas-api-tls\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.191767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.191745 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc1277f-a661-4d8c-8402-4b90554f0ba9-openshift-service-ca\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.292757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.292716 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-maas-api-tls\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.292943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.292779 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc1277f-a661-4d8c-8402-4b90554f0ba9-openshift-service-ca\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.292943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.292856 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2bs\" (UniqueName: \"kubernetes.io/projected/8cc1277f-a661-4d8c-8402-4b90554f0ba9-kube-api-access-fl2bs\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.292943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.292938 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-metrics-certs\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.293649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.293616 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc1277f-a661-4d8c-8402-4b90554f0ba9-openshift-service-ca\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.295964 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.295940 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-maas-api-tls\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.296068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.295976 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-metrics-certs\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.310280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.310227 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2bs\" (UniqueName: \"kubernetes.io/projected/8cc1277f-a661-4d8c-8402-4b90554f0ba9-kube-api-access-fl2bs\") pod \"maas-api-598767f7b6-kstmg\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.440840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.440814 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:03.735569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:03.735486 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-kstmg"] Oct 31 12:35:03.738230 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:35:03.738204 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc1277f_a661_4d8c_8402_4b90554f0ba9.slice/crio-eb6f21d3398d130e79f1d08c9cf5974b454c6fc8068531ad654047f74917f45f WatchSource:0}: Error finding container eb6f21d3398d130e79f1d08c9cf5974b454c6fc8068531ad654047f74917f45f: Status 404 returned error can't find the container with id eb6f21d3398d130e79f1d08c9cf5974b454c6fc8068531ad654047f74917f45f Oct 31 12:35:04.089023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:04.088992 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" event={"ID":"8cc1277f-a661-4d8c-8402-4b90554f0ba9","Type":"ContainerStarted","Data":"eb6f21d3398d130e79f1d08c9cf5974b454c6fc8068531ad654047f74917f45f"} Oct 31 12:35:04.100804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:04.100777 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body= Oct 31 12:35:04.100933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:04.100829 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" Oct 31 12:35:05.100613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:05.100566 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body= Oct 31 12:35:05.101062 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:05.100630 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" Oct 31 12:35:05.189537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:05.189506 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:35:06.100858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:06.100827 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body= Oct 31 12:35:06.101233 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:06.100884 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" Oct 31 12:35:06.243178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:06.243145 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:07.100556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:07.100522 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body= Oct 31 12:35:07.100717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:07.100568 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" Oct 31 12:35:07.105820 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:07.105793 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" event={"ID":"8cc1277f-a661-4d8c-8402-4b90554f0ba9","Type":"ContainerStarted","Data":"21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf"} Oct 31 12:35:07.106156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:07.105867 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:07.126724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:07.126682 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" podStartSLOduration=1.6145792540000001 podStartE2EDuration="4.126665272s" totalImagesPullingTime="2.512086018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:35:07.125594005 +0000 UTC m=+417.940608397" watchObservedRunningTime="2026-10-31 12:35:07.126665272 +0000 UTC m=+417.941679665" Oct 31 12:35:08.104524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:08.104497 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:35:08.105456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:08.105436 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" Oct 31 12:35:13.115615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:13.115591 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 12:35:30.340798 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:35:30.340761 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:35:41.064778 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:41.064744 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:41.218266 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:41.218234 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:45.375615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:45.375583 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:45.508857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:45.508822 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:57.776271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:57.776192 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:58.052336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.052254 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:35:58.764552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.764522 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6"] Oct 31 12:35:58.768120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.768088 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.777525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.777483 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 12:35:58.778193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.778170 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2xp25\"" Oct 31 12:35:58.778193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.778187 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 12:35:58.778457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.778440 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 12:35:58.779213 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.779188 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6"] Oct 31 12:35:58.844487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.844447 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.844582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.844496 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.844582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.844559 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.844662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.844607 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.844662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.844649 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk7bq\" (UniqueName: \"kubernetes.io/projected/17bb9e9e-099a-4392-87da-ed6a36d161ba-kube-api-access-tk7bq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.844737 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.844679 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17bb9e9e-099a-4392-87da-ed6a36d161ba-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.861592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.861573 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:58.945683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.945648 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7bq\" (UniqueName: \"kubernetes.io/projected/17bb9e9e-099a-4392-87da-ed6a36d161ba-kube-api-access-tk7bq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.945838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.945696 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17bb9e9e-099a-4392-87da-ed6a36d161ba-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.945838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.945774 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.945838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.945806 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.945838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.945834 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.946034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.945864 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.946292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.946266 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.946375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.946293 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.946375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.946323 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.948178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.948151 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17bb9e9e-099a-4392-87da-ed6a36d161ba-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.948506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.948489 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17bb9e9e-099a-4392-87da-ed6a36d161ba-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:58.957547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:58.957527 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7bq\" (UniqueName: \"kubernetes.io/projected/17bb9e9e-099a-4392-87da-ed6a36d161ba-kube-api-access-tk7bq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6\" (UID: \"17bb9e9e-099a-4392-87da-ed6a36d161ba\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:59.082750 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:59.082723 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:35:59.361425 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:59.361363 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6"] Oct 31 12:35:59.364126 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:35:59.364100 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bb9e9e_099a_4392_87da_ed6a36d161ba.slice/crio-007776e5547c58d02fd13f6a1f2e5cae888fe0c6d9ecf953daaf34ea97a545da WatchSource:0}: Error finding container 007776e5547c58d02fd13f6a1f2e5cae888fe0c6d9ecf953daaf34ea97a545da: Status 404 returned error can't find the container with id 007776e5547c58d02fd13f6a1f2e5cae888fe0c6d9ecf953daaf34ea97a545da Oct 31 12:35:59.500815 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:35:59.500778 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:36:00.341283 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:36:00.341245 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:36:00.347808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:00.347769 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" event={"ID":"17bb9e9e-099a-4392-87da-ed6a36d161ba","Type":"ContainerStarted","Data":"007776e5547c58d02fd13f6a1f2e5cae888fe0c6d9ecf953daaf34ea97a545da"} Oct 31 12:36:01.119919 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:01.119888 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:01.353363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:01.353331 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" event={"ID":"17bb9e9e-099a-4392-87da-ed6a36d161ba","Type":"ContainerStarted","Data":"133de3c2c96b5d2877813d85798a2af3fd23c9a19b78bf2392f337416687a7bd"} Oct 31 12:36:01.375355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:01.375272 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" podStartSLOduration=1.961665849 podStartE2EDuration="3.375259071s" totalImagesPullingTime="1.413593222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:01.374068161 +0000 UTC m=+472.189082553" watchObservedRunningTime="2026-10-31 12:36:01.375259071 +0000 UTC m=+472.190273461" Oct 31 12:36:01.739521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:01.739433 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:36:02.083327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:02.083294 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:36:03.666956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.666925 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr"] Oct 31 12:36:03.671427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.671403 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.695267 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.695248 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 12:36:03.701238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.701218 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr"] Oct 31 12:36:03.791131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.791110 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.791238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.791137 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.791238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.791163 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa9cd02-a108-4237-bc59-eb14f393cd2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.791238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.791188 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmt6d\" (UniqueName: \"kubernetes.io/projected/bfa9cd02-a108-4237-bc59-eb14f393cd2c-kube-api-access-nmt6d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.791345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.791288 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.791345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.791315 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.822226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.822200 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:03.892443 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892412 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892451 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892544 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892561 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892586 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa9cd02-a108-4237-bc59-eb14f393cd2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892713 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmt6d\" (UniqueName: \"kubernetes.io/projected/bfa9cd02-a108-4237-bc59-eb14f393cd2c-kube-api-access-nmt6d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892860 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892902 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.892947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.892932 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.894795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.894769 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa9cd02-a108-4237-bc59-eb14f393cd2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.895041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.895025 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa9cd02-a108-4237-bc59-eb14f393cd2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.951929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.951875 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmt6d\" (UniqueName: \"kubernetes.io/projected/bfa9cd02-a108-4237-bc59-eb14f393cd2c-kube-api-access-nmt6d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr\" (UID: \"bfa9cd02-a108-4237-bc59-eb14f393cd2c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:03.984618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:03.984598 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:04.258764 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:04.258739 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr"] Oct 31 12:36:04.261172 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:36:04.261145 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa9cd02_a108_4237_bc59_eb14f393cd2c.slice/crio-9d0830277ffb1b7cd98ef0b6d7dac74dc62139f5aaa957d05989515d3fc5d620 WatchSource:0}: Error finding container 9d0830277ffb1b7cd98ef0b6d7dac74dc62139f5aaa957d05989515d3fc5d620: Status 404 returned error can't find the container with id 9d0830277ffb1b7cd98ef0b6d7dac74dc62139f5aaa957d05989515d3fc5d620 Oct 31 12:36:04.368833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:04.368805 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" event={"ID":"bfa9cd02-a108-4237-bc59-eb14f393cd2c","Type":"ContainerStarted","Data":"9d0830277ffb1b7cd98ef0b6d7dac74dc62139f5aaa957d05989515d3fc5d620"} Oct 31 12:36:04.488543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:04.488520 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:36:05.377392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:05.377350 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" event={"ID":"bfa9cd02-a108-4237-bc59-eb14f393cd2c","Type":"ContainerStarted","Data":"4b3feda58b1839d6e40ed3722c21e7a8ffc6fe45eb92baaa0f2ca481b3995871"} Oct 31 12:36:05.985702 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:05.985669 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:06.285411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:06.285329 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:07.589485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:07.589433 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:08.260929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.260860 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" podStartSLOduration=4.975430397 podStartE2EDuration="5.260833359s" totalImagesPullingTime="285.402962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:05.396960684 +0000 UTC m=+476.211975077" watchObservedRunningTime="2026-10-31 12:36:08.260833359 +0000 UTC m=+479.075847752" Oct 31 12:36:08.262214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.262188 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv"] Oct 31 12:36:08.266482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.266440 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.274548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.274531 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 12:36:08.285138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.285116 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv"] Oct 31 12:36:08.372796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.372765 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:08.375516 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.375493 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:36:08.434646 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.434615 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.434646 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.434646 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.434852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.434695 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b468147-6eba-4cbd-aff5-d13da0f35497-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.434852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.434834 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.434954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.434931 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhg7g\" (UniqueName: \"kubernetes.io/projected/3b468147-6eba-4cbd-aff5-d13da0f35497-kube-api-access-xhg7g\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.435014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.434969 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.535909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.535837 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.535909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.535870 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.535909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.535890 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b468147-6eba-4cbd-aff5-d13da0f35497-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.536165 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.536027 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.536165 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.536093 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhg7g\" (UniqueName: \"kubernetes.io/projected/3b468147-6eba-4cbd-aff5-d13da0f35497-kube-api-access-xhg7g\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.536165 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.536126 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.536319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.536209 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.536319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.536296 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.536603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.536574 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.538342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.538318 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b468147-6eba-4cbd-aff5-d13da0f35497-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.538644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.538628 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b468147-6eba-4cbd-aff5-d13da0f35497-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.548563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.548539 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhg7g\" (UniqueName: \"kubernetes.io/projected/3b468147-6eba-4cbd-aff5-d13da0f35497-kube-api-access-xhg7g\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xczlv\" (UID: \"3b468147-6eba-4cbd-aff5-d13da0f35497\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.581343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.581319 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:08.859412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:08.859376 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv"] Oct 31 12:36:08.862226 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:36:08.862193 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b468147_6eba_4cbd_aff5_d13da0f35497.slice/crio-f2c6d3cbf2409a27d4af9675f5e74c408bea283f526416e2ef605096fae7b881 WatchSource:0}: Error finding container f2c6d3cbf2409a27d4af9675f5e74c408bea283f526416e2ef605096fae7b881: Status 404 returned error can't find the container with id f2c6d3cbf2409a27d4af9675f5e74c408bea283f526416e2ef605096fae7b881 Oct 31 12:36:09.083190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.083167 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:36:09.096997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.096975 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:36:09.400232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.399537 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" event={"ID":"3b468147-6eba-4cbd-aff5-d13da0f35497","Type":"ContainerStarted","Data":"cc82bf31525c2c536ff84ceb71edb3bc91ef1d27e483926752bccaaedfd5eb88"} Oct 31 12:36:09.400232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.399666 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" event={"ID":"3b468147-6eba-4cbd-aff5-d13da0f35497","Type":"ContainerStarted","Data":"f2c6d3cbf2409a27d4af9675f5e74c408bea283f526416e2ef605096fae7b881"} Oct 31 12:36:09.418065 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.418036 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" Oct 31 12:36:09.421037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.420987 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" podStartSLOduration=1.164627622 podStartE2EDuration="1.420970092s" totalImagesPullingTime="256.34247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:09.418817961 +0000 UTC m=+480.233832353" watchObservedRunningTime="2026-10-31 12:36:09.420970092 +0000 UTC m=+480.235984484" Oct 31 12:36:09.581752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.581722 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:09.584212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:09.584148 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:11.016417 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.016382 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8"] Oct 31 12:36:11.022900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.022874 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.029424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.029403 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 12:36:11.035499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.035475 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8"] Oct 31 12:36:11.057804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.057766 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.057912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.057813 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.057986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.057915 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.058061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.058033 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.058061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.058058 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.058174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.058087 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7r6h\" (UniqueName: \"kubernetes.io/projected/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-kube-api-access-c7r6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.091435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.091400 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:11.159052 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159019 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159079 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159125 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159201 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159228 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159272 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7r6h\" (UniqueName: \"kubernetes.io/projected/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-kube-api-access-c7r6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159524 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159596 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.159800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.159770 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.162068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.162047 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.162319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.162303 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.174451 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.174407 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7r6h\" (UniqueName: \"kubernetes.io/projected/528d4d47-3dc0-48e4-bb8c-2fa2b23091f2-kube-api-access-c7r6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8\" (UID: \"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.339802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.339775 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:11.628324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.628302 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8"] Oct 31 12:36:11.630924 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:36:11.630901 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528d4d47_3dc0_48e4_bb8c_2fa2b23091f2.slice/crio-b3e41e37f55d96cafce87427d1f16574d6ac1f55dc268f16d34b3c61a858beb9 WatchSource:0}: Error finding container b3e41e37f55d96cafce87427d1f16574d6ac1f55dc268f16d34b3c61a858beb9: Status 404 returned error can't find the container with id b3e41e37f55d96cafce87427d1f16574d6ac1f55dc268f16d34b3c61a858beb9 Oct 31 12:36:11.946223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:11.946177 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:36:12.434324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:12.434284 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" event={"ID":"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2","Type":"ContainerStarted","Data":"5177a3514a7dfb670cbdece419d626c74eb41bd83d89d3d7d7bab240b013c1f5"} Oct 31 12:36:12.434743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:12.434352 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" event={"ID":"528d4d47-3dc0-48e4-bb8c-2fa2b23091f2","Type":"ContainerStarted","Data":"b3e41e37f55d96cafce87427d1f16574d6ac1f55dc268f16d34b3c61a858beb9"} Oct 31 12:36:12.454824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:12.454771 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" podStartSLOduration=2.160120385 podStartE2EDuration="2.454753312s" totalImagesPullingTime="294.632927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:12.453538459 +0000 UTC m=+483.268552853" watchObservedRunningTime="2026-10-31 12:36:12.454753312 +0000 UTC m=+483.269767703" Oct 31 12:36:13.340647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:13.340604 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:13.984927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:13.984887 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:13.997889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:13.997862 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:14.453850 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:14.453825 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" Oct 31 12:36:18.582381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:18.582354 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:18.594430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:18.594408 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:19.475633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:19.475607 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" Oct 31 12:36:21.340434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:21.340401 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:21.352528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:21.352502 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:21.484127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:21.484101 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8" Oct 31 12:36:26.003909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:26.003870 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:26.202916 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:26.202881 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:28.272393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:28.272355 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:29.224716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.224687 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr"] Oct 31 12:36:29.229488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.229453 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.257292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.257260 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr"] Oct 31 12:36:29.317534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317499 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.317889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317568 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.317889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317629 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.317889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317653 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.317889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317699 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.317889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317741 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.317889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317817 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.317889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317857 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.318127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.317928 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl9g2\" (UniqueName: \"kubernetes.io/projected/e5f06403-3135-46e2-bf0e-7a08611c8d0c-kube-api-access-fl9g2\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419159 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419190 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419214 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419314 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419343 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl9g2\" (UniqueName: \"kubernetes.io/projected/e5f06403-3135-46e2-bf0e-7a08611c8d0c-kube-api-access-fl9g2\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419362 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419400 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419434 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419456 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419652 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419715 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419793 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.419985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.419959 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.420046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.420018 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.421894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.421864 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.421894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.421881 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.432361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.432338 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f06403-3135-46e2-bf0e-7a08611c8d0c-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.432551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.432505 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl9g2\" (UniqueName: \"kubernetes.io/projected/e5f06403-3135-46e2-bf0e-7a08611c8d0c-kube-api-access-fl9g2\") pod \"maas-default-gateway-openshift-default-645cf464d7-hp9kr\" (UID: \"e5f06403-3135-46e2-bf0e-7a08611c8d0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.552844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.552767 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:29.833211 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.833182 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr"] Oct 31 12:36:29.835935 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:36:29.835906 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f06403_3135_46e2_bf0e_7a08611c8d0c.slice/crio-278727301ca4efbbb69801d7ef4a02756946c131cba9c15b750f460d267de887 WatchSource:0}: Error finding container 278727301ca4efbbb69801d7ef4a02756946c131cba9c15b750f460d267de887: Status 404 returned error can't find the container with id 278727301ca4efbbb69801d7ef4a02756946c131cba9c15b750f460d267de887 Oct 31 12:36:29.838005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.837969 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:36:29.838063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.838044 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:36:29.838098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:29.838075 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:36:30.355800 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:36:30.355755 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:36:30.516068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:30.516032 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" event={"ID":"e5f06403-3135-46e2-bf0e-7a08611c8d0c","Type":"ContainerStarted","Data":"c059b2f7f4a15a345f2459eee5f237917a131f04674988d6df04bd207d749e93"} Oct 31 12:36:30.516247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:30.516089 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" event={"ID":"e5f06403-3135-46e2-bf0e-7a08611c8d0c","Type":"ContainerStarted","Data":"278727301ca4efbbb69801d7ef4a02756946c131cba9c15b750f460d267de887"} Oct 31 12:36:30.543369 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:30.543315 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podStartSLOduration=1.5433007779999999 podStartE2EDuration="1.543300778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:30.542451857 +0000 UTC m=+501.357466248" watchObservedRunningTime="2026-10-31 12:36:30.543300778 +0000 UTC m=+501.358315169" Oct 31 12:36:30.553817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:30.553792 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:30.555167 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:30.555148 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 12:36:30.555229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:30.555193 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 12:36:31.554044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:31.554010 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 12:36:31.554437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:31.554061 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 12:36:31.761057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:31.761023 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:31.898912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:31.898858 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:32.553406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:32.553369 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 12:36:32.553585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:32.553426 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 12:36:33.554121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:33.554089 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 12:36:33.554548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:33.554145 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 12:36:34.553966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:34.553936 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 12:36:34.554122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:34.554003 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 12:36:35.553216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:35.553181 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 12:36:35.553717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:35.553235 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 12:36:36.553915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:36.553881 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 12:36:36.554302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:36.553929 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 12:36:38.554024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:38.553977 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:36:38.554429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:38.554055 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:36:39.594668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:39.594620 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:36:39.595182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:39.594677 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:36:40.146340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:40.146302 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:40.146898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:40.146867 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:40.147923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:40.147894 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" Oct 31 12:36:44.230378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.230343 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j"] Oct 31 12:36:44.234609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.234582 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.246863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.246838 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j"] Oct 31 12:36:44.394040 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.393994 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67586435-3054-4099-808d-8482a5449e01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394040 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394043 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67586435-3054-4099-808d-8482a5449e01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394064 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394167 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394229 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394319 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394389 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394443 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394423 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.394504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.394441 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxws\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-kube-api-access-srxws\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495435 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495520 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495557 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495581 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxws\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-kube-api-access-srxws\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495627 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67586435-3054-4099-808d-8482a5449e01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495658 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67586435-3054-4099-808d-8482a5449e01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495687 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495722 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.495788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495749 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.496138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.495951 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.496138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.496125 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.496290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.496190 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.496290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.496242 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.496401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.496381 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67586435-3054-4099-808d-8482a5449e01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.497805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.497778 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.498153 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.498132 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67586435-3054-4099-808d-8482a5449e01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.508297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.508264 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.508423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.508400 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxws\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-kube-api-access-srxws\") pod \"maas-default-gateway-openshift-default-645cf464d7-v8c4j\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.549638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.549607 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:44.830704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.830670 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j"] Oct 31 12:36:44.839070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.839030 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:36:44.839161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.839113 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:36:44.839161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:44.839151 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:36:45.591962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:45.591923 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" event={"ID":"67586435-3054-4099-808d-8482a5449e01","Type":"ContainerStarted","Data":"304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54"} Oct 31 12:36:45.592357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:45.591982 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" event={"ID":"67586435-3054-4099-808d-8482a5449e01","Type":"ContainerStarted","Data":"2f06b9a36c59dd6ae599cc6f2f905c7187673bc16d4d03a8388582ceb7b70218"} Oct 31 12:36:45.617517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:45.617453 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podStartSLOduration=1.617440421 podStartE2EDuration="1.617440421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:45.616407803 +0000 UTC m=+516.431422194" watchObservedRunningTime="2026-10-31 12:36:45.617440421 +0000 UTC m=+516.432454812" Oct 31 12:36:46.550727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:46.550689 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:46.552168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:46.552143 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v8c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 12:36:46.552298 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:46.552190 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 12:36:47.550401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:47.550355 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v8c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 12:36:47.550865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:47.550416 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 12:36:48.550687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:48.550656 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v8c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 12:36:48.551141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:48.550709 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 12:36:49.549930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:49.549891 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v8c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 12:36:49.550292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:49.549947 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 12:36:50.550552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:50.550519 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v8c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 12:36:50.550960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:50.550572 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 12:36:51.550672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:51.550631 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v8c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 12:36:51.551064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:51.550701 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 12:36:52.987051 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:52.987017 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:52.987540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:52.987433 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:52.988236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:52.988217 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 12:36:56.300941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:56.300907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:56.803109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:56.803075 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:58.360031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:58.359995 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:58.586298 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:58.586265 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:59.851336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:36:59.851302 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:00.014375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:00.014341 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:00.341282 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:37:00.341247 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:37:06.352608 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:06.352565 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:21.014092 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.014000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:21.156165 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.156133 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:21.629280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.629249 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k"] Oct 31 12:37:21.634159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.634138 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.640696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.640674 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-67d410-openshift-default-dockercfg-p2p5s\"" Oct 31 12:37:21.689046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.689012 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k"] Oct 31 12:37:21.772933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.772889 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f4085-7faa-463d-8ac2-66b99c188160-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.772953 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f4085-7faa-463d-8ac2-66b99c188160-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.773097 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773211 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.773149 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773211 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.773202 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8md\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-kube-api-access-tg8md\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.773247 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.773345 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.773376 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.773439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.773403 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.788162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.788136 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:21.874912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.874878 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.874921 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.874940 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8md\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-kube-api-access-tg8md\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.874964 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875007 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875042 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875078 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875148 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f4085-7faa-463d-8ac2-66b99c188160-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875196 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f4085-7faa-463d-8ac2-66b99c188160-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875364 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875532 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875617 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.875784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875757 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.876002 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.875980 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f4085-7faa-463d-8ac2-66b99c188160-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.877416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.877380 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.877761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.877744 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f4085-7faa-463d-8ac2-66b99c188160-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.897274 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.897215 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8md\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-kube-api-access-tg8md\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.898769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.898752 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-bw46k\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.941079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.941047 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv"] Oct 31 12:37:21.945859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.945839 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:21.947447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.947417 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:21.954120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.954099 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c67de2-openshift-default-dockercfg-psh8p\"" Oct 31 12:37:21.963584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:21.963559 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv"] Oct 31 12:37:22.011361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.007872 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:22.076808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.076776 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.076844 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.076924 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.076961 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463c520-2269-4666-a30c-e6f36b31f750-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.077001 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.077041 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8h2\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-kube-api-access-ks8h2\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.077111 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463c520-2269-4666-a30c-e6f36b31f750-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.077201 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.077260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.077236 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.168971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.168894 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6"] Oct 31 12:37:22.172510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.172493 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.178389 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.178362 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.178561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.178405 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.178882 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.178998 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.178997 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179064 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179096 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463c520-2269-4666-a30c-e6f36b31f750-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179134 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179161 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8h2\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-kube-api-access-ks8h2\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179229 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179275 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463c520-2269-4666-a30c-e6f36b31f750-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.179623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179415 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.180121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.179716 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.180771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.180749 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463c520-2269-4666-a30c-e6f36b31f750-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.181192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.181165 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:22.181815 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.181789 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.182478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.182442 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463c520-2269-4666-a30c-e6f36b31f750-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.184521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.184504 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e6c79d-openshift-default-dockercfg-hwsts\"" Oct 31 12:37:22.191807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.191775 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6"] Oct 31 12:37:22.198970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.198950 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.200695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.200673 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8h2\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-kube-api-access-ks8h2\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.234261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.234092 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k"] Oct 31 12:37:22.236969 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:22.236947 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868f4085_7faa_463d_8ac2_66b99c188160.slice/crio-6329912f0ad7d603684fc732682b2943b384d750973cc4d391d6cc5572932049 WatchSource:0}: Error finding container 6329912f0ad7d603684fc732682b2943b384d750973cc4d391d6cc5572932049: Status 404 returned error can't find the container with id 6329912f0ad7d603684fc732682b2943b384d750973cc4d391d6cc5572932049 Oct 31 12:37:22.239221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.239193 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.239279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.239252 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.239313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.239282 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.261444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.261418 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:22.275496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.275453 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988"] Oct 31 12:37:22.279373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.279349 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.279966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.279942 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.279999 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.280068 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmnhc\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-kube-api-access-cmnhc\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.280202 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.280253 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.280311 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.280350 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.280408 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.280541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.280493 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8199b71d-2f24-4d71-99c2-0282fdc31fca-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.295717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.295695 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b25d3b-openshift-default-dockercfg-5r5zv\"" Oct 31 12:37:22.302306 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.302280 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988"] Oct 31 12:37:22.351725 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.351687 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:22.381371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381329 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8199b71d-2f24-4d71-99c2-0282fdc31fca-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.381548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381399 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xph7n\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-kube-api-access-xph7n\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.381548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381430 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.381548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381459 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.381548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381512 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.381548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381536 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.381828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381622 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.381828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381676 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.381828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381704 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.381828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381735 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.381828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381766 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmnhc\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-kube-api-access-cmnhc\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.381828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381819 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.382134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381853 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.382134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381897 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.382134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381933 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.382134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381957 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.382134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.381989 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.382134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.382065 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.382556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.382527 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.382857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.382827 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.383100 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.383075 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.383185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.383157 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.383746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.383723 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8199b71d-2f24-4d71-99c2-0282fdc31fca-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.384955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.384935 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.385922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.385904 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.397982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.397915 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.399309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.399288 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmnhc\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-kube-api-access-cmnhc\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.483448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483355 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xph7n\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-kube-api-access-xph7n\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.483448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483394 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.483448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483414 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.483761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483550 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.483761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483667 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.483884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483773 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.483884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483801 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.483884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483858 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.484041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483915 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.484041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.483922 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.484306 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.484264 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.484418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.484333 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.484509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.484482 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.484578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.484529 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.486417 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.486395 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.486417 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.486407 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.487581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.487404 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:22.504031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.504006 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.505421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.505400 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph7n\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-kube-api-access-xph7n\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.507961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.507939 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:22.568926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.568896 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv"] Oct 31 12:37:22.572862 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:22.572826 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0463c520_2269_4666_a30c_e6f36b31f750.slice/crio-9a2780db906199e966eceb26a85f755b4263a5b843d017a362ef5288878d8496 WatchSource:0}: Error finding container 9a2780db906199e966eceb26a85f755b4263a5b843d017a362ef5288878d8496: Status 404 returned error can't find the container with id 9a2780db906199e966eceb26a85f755b4263a5b843d017a362ef5288878d8496 Oct 31 12:37:22.575544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.575503 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.575663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.575588 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.575663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.575640 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.597908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.597884 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:22.642434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.642399 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:22.774241 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.774150 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" event={"ID":"868f4085-7faa-463d-8ac2-66b99c188160","Type":"ContainerStarted","Data":"b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db"} Oct 31 12:37:22.774241 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.774221 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" event={"ID":"868f4085-7faa-463d-8ac2-66b99c188160","Type":"ContainerStarted","Data":"6329912f0ad7d603684fc732682b2943b384d750973cc4d391d6cc5572932049"} Oct 31 12:37:22.777796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.777721 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6"] Oct 31 12:37:22.780609 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:22.780576 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8199b71d_2f24_4d71_99c2_0282fdc31fca.slice/crio-aa63d2e7b539157c5c9dc3054daa9359d84bebd803680c6d03867a929237e747 WatchSource:0}: Error finding container aa63d2e7b539157c5c9dc3054daa9359d84bebd803680c6d03867a929237e747: Status 404 returned error can't find the container with id aa63d2e7b539157c5c9dc3054daa9359d84bebd803680c6d03867a929237e747 Oct 31 12:37:22.781204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.781175 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" event={"ID":"0463c520-2269-4666-a30c-e6f36b31f750","Type":"ContainerStarted","Data":"78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53"} Oct 31 12:37:22.781316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.781228 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" event={"ID":"0463c520-2269-4666-a30c-e6f36b31f750","Type":"ContainerStarted","Data":"9a2780db906199e966eceb26a85f755b4263a5b843d017a362ef5288878d8496"} Oct 31 12:37:22.782817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.782782 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.782931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.782862 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.782931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.782906 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.802855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.802811 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podStartSLOduration=1.802797626 podStartE2EDuration="1.802797626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:22.80143379 +0000 UTC m=+553.616448182" watchObservedRunningTime="2026-10-31 12:37:22.802797626 +0000 UTC m=+553.617812017" Oct 31 12:37:22.830440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.830392 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podStartSLOduration=1.8303779470000001 podStartE2EDuration="1.830377947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:22.830016936 +0000 UTC m=+553.645031330" watchObservedRunningTime="2026-10-31 12:37:22.830377947 +0000 UTC m=+553.645392338" Oct 31 12:37:22.884977 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.884928 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988"] Oct 31 12:37:22.888187 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:22.888154 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717c9dd9_85aa_4ea6_8510_a40ef7e2af57.slice/crio-e5deef60ac4d67f8b30dd0884e2d3722b1186c2fe7154fb43d7aed3e3e22c40f WatchSource:0}: Error finding container e5deef60ac4d67f8b30dd0884e2d3722b1186c2fe7154fb43d7aed3e3e22c40f: Status 404 returned error can't find the container with id e5deef60ac4d67f8b30dd0884e2d3722b1186c2fe7154fb43d7aed3e3e22c40f Oct 31 12:37:22.892711 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.892670 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.892814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.892757 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.892814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.892803 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:22.959716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:22.955819 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:23.262539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.262505 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:23.267339 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.267304 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:23.788556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.788509 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" event={"ID":"717c9dd9-85aa-4ea6-8510-a40ef7e2af57","Type":"ContainerStarted","Data":"02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db"} Oct 31 12:37:23.788743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.788577 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" event={"ID":"717c9dd9-85aa-4ea6-8510-a40ef7e2af57","Type":"ContainerStarted","Data":"e5deef60ac4d67f8b30dd0884e2d3722b1186c2fe7154fb43d7aed3e3e22c40f"} Oct 31 12:37:23.790073 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.790044 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" event={"ID":"8199b71d-2f24-4d71-99c2-0282fdc31fca","Type":"ContainerStarted","Data":"00b190882d07c3100ac0711518918c9bd8a9d0e685b67276fb6b85a9314a16d7"} Oct 31 12:37:23.790199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.790099 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" event={"ID":"8199b71d-2f24-4d71-99c2-0282fdc31fca","Type":"ContainerStarted","Data":"aa63d2e7b539157c5c9dc3054daa9359d84bebd803680c6d03867a929237e747"} Oct 31 12:37:23.790437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.790414 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:23.791427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.791408 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:23.815532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.815487 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podStartSLOduration=1.815455025 podStartE2EDuration="1.815455025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:23.814321378 +0000 UTC m=+554.629335802" watchObservedRunningTime="2026-10-31 12:37:23.815455025 +0000 UTC m=+554.630469416" Oct 31 12:37:23.874242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.874186 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podStartSLOduration=1.87414683 podStartE2EDuration="1.87414683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:23.871267253 +0000 UTC m=+554.686281644" watchObservedRunningTime="2026-10-31 12:37:23.87414683 +0000 UTC m=+554.689161217" Oct 31 12:37:23.953184 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:23.953153 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:24.488552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.488516 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:24.493872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.493847 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:24.598290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.598261 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:24.603733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.603709 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:24.795271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.795188 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:24.795271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.795248 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:24.795271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.795272 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:24.795762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.795741 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:24.795970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.795953 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:24.796036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:24.796025 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:25.113259 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:25.113229 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:25.268384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:25.268329 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:25.416775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:25.416667 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:25.722952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:25.722873 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:25.924684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:25.924651 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:26.101668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:26.101635 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:26.386777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:26.386695 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:26.581512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:26.573926 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:26.730453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:26.730340 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:26.889569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:26.889541 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:27.100767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:27.100736 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:27.299878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:27.299836 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:27.479543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:27.478697 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:27.840974 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:27.840940 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.095881 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:28.095787 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.500301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:28.500218 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.650200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:28.650164 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.987944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:28.987906 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:29.025610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.025575 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr"] Oct 31 12:37:29.029672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.029650 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.036799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.036759 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b218d3\"" Oct 31 12:37:29.040132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.040109 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b218d3-dockercfg-zfvtp\"" Oct 31 12:37:29.041625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.041601 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr"] Oct 31 12:37:29.059297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.059265 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.059415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.059325 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9s4\" (UniqueName: \"kubernetes.io/projected/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-kube-api-access-gr9s4\") pod \"payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.063344 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.063319 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6"] Oct 31 12:37:29.067547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.067525 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.082899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.082875 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6"] Oct 31 12:37:29.139275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.139248 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-b25d3b-serving-cert\"" Oct 31 12:37:29.143333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.143306 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:29.160832 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.160801 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9s4\" (UniqueName: \"kubernetes.io/projected/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-kube-api-access-gr9s4\") pod \"payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.160984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.160852 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.160984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.160966 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-kube-api-access-x8z4w\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.161116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.161100 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-metrics-certs\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.161151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.161128 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.161283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.161264 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.161984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.161965 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.178842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.178818 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9s4\" (UniqueName: \"kubernetes.io/projected/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-kube-api-access-gr9s4\") pod \"payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.262410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.262318 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.262410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.262362 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-kube-api-access-x8z4w\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.262669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.262428 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-metrics-certs\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.262669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.262449 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.263234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.263213 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.265871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.265843 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-metrics-certs\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.273003 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.272976 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.277024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.277002 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-kube-api-access-x8z4w\") pod \"maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.287917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.287893 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:29.344714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.344691 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:29.386045 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.386018 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:29.450871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.450838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:29.619668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.619646 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr"] Oct 31 12:37:29.621978 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:29.621950 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8a5e56_cb16_48ce_afcc_d05ba8efb79c.slice/crio-8a71b534730e552e0b90bd11e18da6d760e1a9bc7aaed196ad553cc05c841a57 WatchSource:0}: Error finding container 8a71b534730e552e0b90bd11e18da6d760e1a9bc7aaed196ad553cc05c841a57: Status 404 returned error can't find the container with id 8a71b534730e552e0b90bd11e18da6d760e1a9bc7aaed196ad553cc05c841a57 Oct 31 12:37:29.623915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.623900 6727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:37:29.690803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.690780 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6"] Oct 31 12:37:29.693367 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:29.693335 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6faedd5_b9c9_47b6_bcdd_d2c5729f9fc0.slice/crio-b00ff065afbed39a3917b70bc60a83dc557c441adc7c3756a07a0cc04f92e7b9 WatchSource:0}: Error finding container b00ff065afbed39a3917b70bc60a83dc557c441adc7c3756a07a0cc04f92e7b9: Status 404 returned error can't find the container with id b00ff065afbed39a3917b70bc60a83dc557c441adc7c3756a07a0cc04f92e7b9 Oct 31 12:37:29.822024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.821970 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" event={"ID":"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0","Type":"ContainerStarted","Data":"b00ff065afbed39a3917b70bc60a83dc557c441adc7c3756a07a0cc04f92e7b9"} Oct 31 12:37:29.823406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:29.823368 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" event={"ID":"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c","Type":"ContainerStarted","Data":"8a71b534730e552e0b90bd11e18da6d760e1a9bc7aaed196ad553cc05c841a57"} Oct 31 12:37:30.033998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:30.033915 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:30.206615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:30.206519 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:30.345959 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:37:30.345922 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:37:30.771806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:30.771718 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:30.831140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:30.831095 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" event={"ID":"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0","Type":"ContainerStarted","Data":"39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c"} Oct 31 12:37:30.831349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:30.831331 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:31.033188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.033093 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:31.068297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.068233 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" podStartSLOduration=1.636766849 podStartE2EDuration="2.068213943s" totalImagesPullingTime="431.447094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:30.89875036 +0000 UTC m=+561.713764752" watchObservedRunningTime="2026-10-31 12:37:31.068213943 +0000 UTC m=+561.883228335" Oct 31 12:37:31.069068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.069037 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:37:31.165190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.165158 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9"] Oct 31 12:37:31.175162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.175136 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.181884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.181859 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9"] Oct 31 12:37:31.190233 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.190204 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e6c79d-serving-cert\"" Oct 31 12:37:31.220156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.220128 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.220294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.220166 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-metrics-certs\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.220294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.220278 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr74c\" (UniqueName: \"kubernetes.io/projected/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-kube-api-access-gr74c\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.220459 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.220435 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.241291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.241258 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:31.322776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.321960 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.322776 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:37:31.322056 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-serving-cert: secret "maas-api-e2e-worker-w2-e6c79d-serving-cert" not found Oct 31 12:37:31.322776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.322137 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.322776 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:37:31.322185 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls podName:50f9d07b-5a33-48d1-aa2a-db08ed4e5882 nodeName:}" failed. No retries permitted until 2026-10-31 12:37:31.82215594 +0000 UTC m=+562.637170319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls") pod "maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" (UID: "50f9d07b-5a33-48d1-aa2a-db08ed4e5882") : secret "maas-api-e2e-worker-w2-e6c79d-serving-cert" not found Oct 31 12:37:31.322776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.322237 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-metrics-certs\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.322776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.322298 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr74c\" (UniqueName: \"kubernetes.io/projected/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-kube-api-access-gr74c\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.322776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.322755 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.325577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.325553 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-metrics-certs\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.346340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.346312 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr74c\" (UniqueName: \"kubernetes.io/projected/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-kube-api-access-gr74c\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.386416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.386380 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:31.561485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.557920 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:31.771348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.771264 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:31.830025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.829987 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:31.832493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:31.832449 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:32.102192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.102158 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:32.345402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.345346 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:32.398595 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.398442 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9"] Oct 31 12:37:32.411851 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:32.411818 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f9d07b_5a33_48d1_aa2a_db08ed4e5882.slice/crio-d0977e6b76574b0c0c341c71c6d7fc95524bf487b021b3097b7ebf712404d2de WatchSource:0}: Error finding container d0977e6b76574b0c0c341c71c6d7fc95524bf487b021b3097b7ebf712404d2de: Status 404 returned error can't find the container with id d0977e6b76574b0c0c341c71c6d7fc95524bf487b021b3097b7ebf712404d2de Oct 31 12:37:32.485361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.485313 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:32.635698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.635655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:32.843835 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.843804 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" event={"ID":"50f9d07b-5a33-48d1-aa2a-db08ed4e5882","Type":"ContainerStarted","Data":"d0977e6b76574b0c0c341c71c6d7fc95524bf487b021b3097b7ebf712404d2de"} Oct 31 12:37:32.846230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.846211 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" event={"ID":"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c","Type":"ContainerStarted","Data":"98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9"} Oct 31 12:37:32.846895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.846876 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:32.889922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:32.889855 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" podStartSLOduration=2.659488315 podStartE2EDuration="4.889835844s" totalImagesPullingTime="2.230347529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:32.888953286 +0000 UTC m=+563.703967679" watchObservedRunningTime="2026-10-31 12:37:32.889835844 +0000 UTC m=+563.704850236" Oct 31 12:37:33.097029 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.096941 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw"] Oct 31 12:37:33.100996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.100975 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.109837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.109810 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-85ce08-serving-cert\"" Oct 31 12:37:33.117627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.117607 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw"] Oct 31 12:37:33.145972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.145901 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.146112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.146050 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.146112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.146087 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwn9\" (UniqueName: \"kubernetes.io/projected/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-kube-api-access-4zwn9\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.146112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.146109 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-metrics-certs\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.190129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.190097 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:33.247259 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.247215 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.247395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.247328 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.247395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.247356 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwn9\" (UniqueName: \"kubernetes.io/projected/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-kube-api-access-4zwn9\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.247395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.247383 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-metrics-certs\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.248074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.248050 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.249690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.249661 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-metrics-certs\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.249790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.249730 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.259704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.259683 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwn9\" (UniqueName: \"kubernetes.io/projected/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-kube-api-access-4zwn9\") pod \"maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.340611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.340529 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:33.416941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.416852 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:33.722450 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.722371 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw"] Oct 31 12:37:33.727514 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:33.727353 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8dd1bb_ee38_4ebb_a594_ccbbef643fa3.slice/crio-c32aefcad6c20db356ddb73b00b424b36c1a01da03284b63fdd0d703385a3502 WatchSource:0}: Error finding container c32aefcad6c20db356ddb73b00b424b36c1a01da03284b63fdd0d703385a3502: Status 404 returned error can't find the container with id c32aefcad6c20db356ddb73b00b424b36c1a01da03284b63fdd0d703385a3502 Oct 31 12:37:33.852930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.852893 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" event={"ID":"50f9d07b-5a33-48d1-aa2a-db08ed4e5882","Type":"ContainerStarted","Data":"c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4"} Oct 31 12:37:33.853113 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.852963 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:33.854383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:33.854355 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" event={"ID":"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3","Type":"ContainerStarted","Data":"c32aefcad6c20db356ddb73b00b424b36c1a01da03284b63fdd0d703385a3502"} Oct 31 12:37:34.089486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.079227 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" podStartSLOduration=2.650600318 podStartE2EDuration="3.079204262s" totalImagesPullingTime="428.603944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:33.887767666 +0000 UTC m=+564.702782058" watchObservedRunningTime="2026-10-31 12:37:34.079204262 +0000 UTC m=+564.894218656" Oct 31 12:37:34.098079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.095533 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj"] Oct 31 12:37:34.101562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.101535 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj"] Oct 31 12:37:34.101681 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.101659 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.105342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.105319 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c67de2-serving-cert\"" Oct 31 12:37:34.157933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.157904 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.158279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.157950 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grhnr\" (UniqueName: \"kubernetes.io/projected/10534e17-bdff-4137-a4b1-8cac04c80b73-kube-api-access-grhnr\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.158279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.158088 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10534e17-bdff-4137-a4b1-8cac04c80b73-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.158279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.158199 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-metrics-certs\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.187807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.187775 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:34.259626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.259604 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.259749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.259638 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grhnr\" (UniqueName: \"kubernetes.io/projected/10534e17-bdff-4137-a4b1-8cac04c80b73-kube-api-access-grhnr\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.259749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.259696 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10534e17-bdff-4137-a4b1-8cac04c80b73-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.259875 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:37:34.259766 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-serving-cert: secret "maas-api-e2e-worker-w4-c67de2-serving-cert" not found Oct 31 12:37:34.259875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.259774 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-metrics-certs\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.259875 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:37:34.259831 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls podName:10534e17-bdff-4137-a4b1-8cac04c80b73 nodeName:}" failed. No retries permitted until 2026-10-31 12:37:34.759810977 +0000 UTC m=+565.574825346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls") pod "maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" (UID: "10534e17-bdff-4137-a4b1-8cac04c80b73") : secret "maas-api-e2e-worker-w4-c67de2-serving-cert" not found Oct 31 12:37:34.260458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.260432 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10534e17-bdff-4137-a4b1-8cac04c80b73-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.262438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.262419 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-metrics-certs\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.288246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.288125 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grhnr\" (UniqueName: \"kubernetes.io/projected/10534e17-bdff-4137-a4b1-8cac04c80b73-kube-api-access-grhnr\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.332878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.332853 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:34.526779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.526705 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:34.792283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.792187 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.795508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.795171 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:34.861578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.861544 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" event={"ID":"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3","Type":"ContainerStarted","Data":"9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1"} Oct 31 12:37:34.861840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.861821 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:34.887512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.886593 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" podStartSLOduration=1.3963834450000001 podStartE2EDuration="1.886575103s" totalImagesPullingTime="490.191658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:34.884812939 +0000 UTC m=+565.699827334" watchObservedRunningTime="2026-10-31 12:37:34.886575103 +0000 UTC m=+565.701589495" Oct 31 12:37:34.950454 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:34.950433 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:35.018685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.018652 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:35.318766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.318742 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj"] Oct 31 12:37:35.320979 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:37:35.320948 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10534e17_bdff_4137_a4b1_8cac04c80b73.slice/crio-4000a7dbc4bbf43a884c618806e0dc74516a4440016cb22f52237ad7827e2c2a WatchSource:0}: Error finding container 4000a7dbc4bbf43a884c618806e0dc74516a4440016cb22f52237ad7827e2c2a: Status 404 returned error can't find the container with id 4000a7dbc4bbf43a884c618806e0dc74516a4440016cb22f52237ad7827e2c2a Oct 31 12:37:35.751769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.751734 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:35.914489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.913345 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:35.918604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.918578 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" event={"ID":"10534e17-bdff-4137-a4b1-8cac04c80b73","Type":"ContainerStarted","Data":"8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132"} Oct 31 12:37:35.918722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.918620 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" event={"ID":"10534e17-bdff-4137-a4b1-8cac04c80b73","Type":"ContainerStarted","Data":"4000a7dbc4bbf43a884c618806e0dc74516a4440016cb22f52237ad7827e2c2a"} Oct 31 12:37:35.918722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.918656 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:35.939891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.939843 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" podStartSLOduration=1.509845307 podStartE2EDuration="1.939825895s" totalImagesPullingTime="429.980588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:35.938503564 +0000 UTC m=+566.753517949" watchObservedRunningTime="2026-10-31 12:37:35.939825895 +0000 UTC m=+566.754840288" Oct 31 12:37:35.997083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:35.997045 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:37:36.605887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:36.605854 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:36.844022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:36.843994 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:37:38.186488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:38.185442 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:39.313488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:39.311106 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:39.945832 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:39.945805 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:37:40.254680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:40.254577 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:40.849633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:40.849598 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:37:40.849981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:40.849659 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:37:40.947768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:40.947733 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:37:41.190542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:41.187190 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:41.294184 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:41.294142 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:37:41.528711 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:41.528573 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:41.711991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:41.711915 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:41.857862 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:41.857830 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:42.983747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:42.983712 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 12:37:43.104446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:43.104352 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:44.313902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:44.313849 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:44.694808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:44.694775 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:44.866238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:44.866179 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:37:45.235196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:45.235154 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:45.372733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:45.372690 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:45.538566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:45.538448 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:45.764903 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:45.764866 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:45.951097 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:45.951059 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:46.102275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:46.102245 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:46.348225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:46.348196 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:46.798588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:46.798509 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.096732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:46.968808 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.096732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:47.161594 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.096732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:47.836998 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.096732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:48.527429 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.261851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.261825 6727 generic.go:356] "Generic (PLEG): container finished" podID="0463c520-2269-4666-a30c-e6f36b31f750" containerID="78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53" exitCode=0 Oct 31 12:37:49.262061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.261895 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" event={"ID":"0463c520-2269-4666-a30c-e6f36b31f750","Type":"ContainerDied","Data":"78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53"} Oct 31 12:37:49.262308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.262292 6727 scope.go:122] "RemoveContainer" containerID="78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53" Oct 31 12:37:49.263791 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.263767 6727 generic.go:356] "Generic (PLEG): container finished" podID="868f4085-7faa-463d-8ac2-66b99c188160" containerID="b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db" exitCode=0 Oct 31 12:37:49.263894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.263851 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.263894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.263876 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" event={"ID":"868f4085-7faa-463d-8ac2-66b99c188160","Type":"ContainerDied","Data":"b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db"} Oct 31 12:37:49.263972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.263930 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.264006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.263978 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.264267 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.264248 6727 scope.go:122] "RemoveContainer" containerID="b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db" Oct 31 12:37:49.265748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.265717 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.265858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.265806 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.265924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.265861 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.265979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.265911 6727 generic.go:356] "Generic (PLEG): container finished" podID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerID="02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db" exitCode=0 Oct 31 12:37:49.266038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.265984 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" event={"ID":"717c9dd9-85aa-4ea6-8510-a40ef7e2af57","Type":"ContainerDied","Data":"02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db"} Oct 31 12:37:49.266337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.266321 6727 scope.go:122] "RemoveContainer" containerID="02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db" Oct 31 12:37:49.267798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.267591 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.267798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.267656 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.267798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.267694 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.268528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.268506 6727 generic.go:356] "Generic (PLEG): container finished" podID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerID="00b190882d07c3100ac0711518918c9bd8a9d0e685b67276fb6b85a9314a16d7" exitCode=0 Oct 31 12:37:49.268622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.268581 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" event={"ID":"8199b71d-2f24-4d71-99c2-0282fdc31fca","Type":"ContainerDied","Data":"00b190882d07c3100ac0711518918c9bd8a9d0e685b67276fb6b85a9314a16d7"} Oct 31 12:37:49.268929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.268914 6727 scope.go:122] "RemoveContainer" containerID="00b190882d07c3100ac0711518918c9bd8a9d0e685b67276fb6b85a9314a16d7" Oct 31 12:37:49.273337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.273309 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.273437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.273383 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.273555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.273433 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:37:49.308898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.308850 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.663805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:49.663694 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:37:50.314502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.313698 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" event={"ID":"717c9dd9-85aa-4ea6-8510-a40ef7e2af57","Type":"ContainerStarted","Data":"edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20"} Oct 31 12:37:50.322492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.318441 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:50.347485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.345888 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" event={"ID":"8199b71d-2f24-4d71-99c2-0282fdc31fca","Type":"ContainerStarted","Data":"df17183ff35e66b4ff7c6c5a122cb02d832966a4c9c12e2ebcd0add1a66d6704"} Oct 31 12:37:50.371482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.371125 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" event={"ID":"0463c520-2269-4666-a30c-e6f36b31f750","Type":"ContainerStarted","Data":"6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390"} Oct 31 12:37:50.402344 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.401952 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" event={"ID":"868f4085-7faa-463d-8ac2-66b99c188160","Type":"ContainerStarted","Data":"64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794"} Oct 31 12:37:50.454857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.454823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:50.495983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.495900 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:50.497579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.497545 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:37:50.497812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.497789 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:37:50.600656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.600624 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:50.601031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.601003 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:37:50.601207 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.601180 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:37:50.900518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.900400 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:50.949830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.949795 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:50.951570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.951541 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:37:50.951784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:50.951757 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:37:51.065488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.053717 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:51.203990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.203904 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:51.270730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.270700 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:51.271097 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.271073 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:37:51.271244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.271221 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:37:51.434079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.434048 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:51.489872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.489787 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:37:51.490117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.490090 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:37:51.619487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.608947 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:37:51.619487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.609016 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:37:51.948926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.948890 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:37:51.949241 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:51.949193 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:37:52.049487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.047683 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:52.204551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.203575 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:52.286487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.268917 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:37:52.286487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.268997 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:37:52.530545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.521504 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:52.530545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.523608 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:37:52.530545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.523658 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:37:52.627501 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.602028 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:37:52.627501 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.602100 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:37:52.726569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.726532 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:52.965488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.961653 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:37:52.965488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:52.961722 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:37:53.264118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.264011 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:37:53.264432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.264399 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:37:53.493379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.491570 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:37:53.493379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.491631 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:37:53.599000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.598962 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:37:53.599731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.599623 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:37:53.731499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.728839 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:53.790391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.790362 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:53.948083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.947981 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:37:53.948368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:53.948321 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:37:54.266541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.263861 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:37:54.266541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.263925 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:37:54.512486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.488316 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:37:54.512486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.491917 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:37:54.601898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.601862 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:37:54.606485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.602442 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:37:54.802488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.798569 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:54.802488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.799531 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:54.802488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.800241 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:54.960544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.953848 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:37:54.960544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:54.953907 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:37:55.302726 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.302604 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:55.303093 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.303066 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:37:55.303283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.303256 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:37:55.489889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.489841 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:37:55.490188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.490152 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:37:55.608910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.608869 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:37:55.609568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.609525 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:37:55.769488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:55.768540 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:56.030546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:56.029960 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:37:56.030546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:56.030027 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:37:56.261909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:56.261870 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:37:56.262216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:56.262186 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:37:56.488227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:56.488194 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:37:56.488415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:56.488253 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:37:56.958444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:56.958406 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:57.141168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.141109 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:57.271410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.270069 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:57.272567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.272539 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:37:57.305046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.305021 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:57.306839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.306820 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:37:57.414588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.414557 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:57.416174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.416154 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:37:57.492096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.492059 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:57.493586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.493562 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:37:57.841714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:57.841682 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:58.624600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:58.624570 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:59.323536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:37:59.323507 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:00.111603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:00.111574 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:00.342450 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:38:00.342414 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:38:00.818790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:00.818761 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:00.941821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:00.941785 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:38:01.443193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.443163 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv"] Oct 31 12:38:01.449436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.449415 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.460859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.460837 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"default-dockercfg-5hqp9\"" Oct 31 12:38:01.461122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.461102 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"simulator-w0-fb0343-kserve-self-signed-certs\"" Oct 31 12:38:01.462393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.462373 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"openshift-service-ca.crt\"" Oct 31 12:38:01.462393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.462384 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"kube-root-ca.crt\"" Oct 31 12:38:01.463208 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.463177 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv"] Oct 31 12:38:01.532011 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.531979 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tmp-dir\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.532122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.532041 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tls-certs\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.532173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.532155 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-dshm\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.532217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.532204 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-home\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.532320 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.532304 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-model-cache\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.532407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.532392 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrw4\" (UniqueName: \"kubernetes.io/projected/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-kube-api-access-lvrw4\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.537006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.536985 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:01.634100 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.634074 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrw4\" (UniqueName: \"kubernetes.io/projected/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-kube-api-access-lvrw4\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.634214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.634127 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tmp-dir\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.634214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.634183 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tls-certs\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.634334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.634241 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-dshm\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.634334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.634296 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-home\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.634434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.634358 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-model-cache\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.634864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.634836 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-model-cache\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.635181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.635150 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-home\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.635544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.635515 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tmp-dir\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.637105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.637081 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-dshm\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.637391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.637366 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tls-certs\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.668299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.668236 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrw4\" (UniqueName: \"kubernetes.io/projected/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-kube-api-access-lvrw4\") pod \"simulator-w0-fb0343-kserve-85557484b7-w8gkv\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.761566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.761483 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:01.947806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:01.947691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:02.059707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:02.059683 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv"] Oct 31 12:38:02.061686 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:38:02.061658 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65c629cd_c1e0_4fbd_8b03_1eea66a6dc16.slice/crio-8a5a40386b52f5edb437f140981a1dc6ef62539b2cad2188320db1c61399706a WatchSource:0}: Error finding container 8a5a40386b52f5edb437f140981a1dc6ef62539b2cad2188320db1c61399706a: Status 404 returned error can't find the container with id 8a5a40386b52f5edb437f140981a1dc6ef62539b2cad2188320db1c61399706a Oct 31 12:38:02.716758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:02.716726 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:02.822574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:02.822534 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" event={"ID":"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16","Type":"ContainerStarted","Data":"4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d"} Oct 31 12:38:02.822768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:02.822587 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" event={"ID":"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16","Type":"ContainerStarted","Data":"8a5a40386b52f5edb437f140981a1dc6ef62539b2cad2188320db1c61399706a"} Oct 31 12:38:02.841101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:02.841049 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podStartSLOduration=1.841033218 podStartE2EDuration="1.841033218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:02.840769309 +0000 UTC m=+593.655783700" watchObservedRunningTime="2026-10-31 12:38:02.841033218 +0000 UTC m=+593.656047611" Oct 31 12:38:03.432369 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:03.432326 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:03.761788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:03.761719 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:03.812252 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:03.812214 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:04.236103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:04.236070 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:04.920228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:04.920197 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:05.622512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:05.622473 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:05.909088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:05.909014 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:06.344646 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:06.344614 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:06.480605 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:06.480572 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:07.157863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:07.157834 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:08.027842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:08.027814 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:08.722232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:08.722202 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:09.439035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:09.439001 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:10.155757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:10.155722 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:10.810363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:10.810330 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:11.512847 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:11.512814 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:11.761948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:11.761915 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:11.774598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:11.774530 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:11.881018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:11.880992 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:38:12.213083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:12.213052 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:12.926223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:12.926191 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:13.226019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.225939 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj"] Oct 31 12:38:13.231229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.231207 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.238747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.238702 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"kube-root-ca.crt\"" Oct 31 12:38:13.238909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.238876 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"premium-w5-b25d3b-kserve-self-signed-certs\"" Oct 31 12:38:13.239008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.238955 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"default-dockercfg-rt29c\"" Oct 31 12:38:13.239128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.239112 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"openshift-service-ca.crt\"" Oct 31 12:38:13.241617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.241595 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj"] Oct 31 12:38:13.310706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.310668 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:13.364024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.363996 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-home\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.364144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.364031 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-tmp-dir\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.364144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.364111 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-model-cache\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.364318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.364149 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07021190-faf9-432c-b168-573737c7a3c4-tls-certs\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.364318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.364296 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-dshm\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.364423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.364366 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27fg\" (UniqueName: \"kubernetes.io/projected/07021190-faf9-432c-b168-573737c7a3c4-kube-api-access-z27fg\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465045 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27fg\" (UniqueName: \"kubernetes.io/projected/07021190-faf9-432c-b168-573737c7a3c4-kube-api-access-z27fg\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465110 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-home\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465127 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-tmp-dir\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465154 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-model-cache\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465179 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07021190-faf9-432c-b168-573737c7a3c4-tls-certs\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465255 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-dshm\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465668 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-home\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465791 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-tmp-dir\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.465880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.465827 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-model-cache\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.467923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.467898 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-dshm\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.468061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.468044 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07021190-faf9-432c-b168-573737c7a3c4-tls-certs\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.477459 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.477404 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27fg\" (UniqueName: \"kubernetes.io/projected/07021190-faf9-432c-b168-573737c7a3c4-kube-api-access-z27fg\") pod \"premium-w5-b25d3b-kserve-849bd65849-xzdpj\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.535104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.535077 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:38:13.545613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.545593 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:13.749628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.749519 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:13.820772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.820749 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj"] Oct 31 12:38:13.822748 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:38:13.822718 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07021190_faf9_432c_b168_573737c7a3c4.slice/crio-93f5aca03c7036221894f9a4ec967ce634307c1591662efdc9d5b4e053a42914 WatchSource:0}: Error finding container 93f5aca03c7036221894f9a4ec967ce634307c1591662efdc9d5b4e053a42914: Status 404 returned error can't find the container with id 93f5aca03c7036221894f9a4ec967ce634307c1591662efdc9d5b4e053a42914 Oct 31 12:38:13.881850 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.881821 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" event={"ID":"07021190-faf9-432c-b168-573737c7a3c4","Type":"ContainerStarted","Data":"e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163"} Oct 31 12:38:13.881951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.881862 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" event={"ID":"07021190-faf9-432c-b168-573737c7a3c4","Type":"ContainerStarted","Data":"93f5aca03c7036221894f9a4ec967ce634307c1591662efdc9d5b4e053a42914"} Oct 31 12:38:13.922123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:13.922080 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podStartSLOduration=0.922065281 podStartE2EDuration="922.065281ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:13.92023349 +0000 UTC m=+604.735247880" watchObservedRunningTime="2026-10-31 12:38:13.922065281 +0000 UTC m=+604.737079671" Oct 31 12:38:14.546060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:14.546027 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:14.626307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:14.626279 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:15.330372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:15.330342 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:16.021043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:16.021010 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:16.712714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:16.712680 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:17.433715 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:17.433674 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:18.136630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:18.136599 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:18.830184 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:18.830157 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:19.608781 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:19.608732 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:20.338822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.338789 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:20.623627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.623563 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb"] Oct 31 12:38:20.627677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.627657 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.636117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.636087 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"premium-w6-67d410-kserve-self-signed-certs\"" Oct 31 12:38:20.636245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.636145 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"kube-root-ca.crt\"" Oct 31 12:38:20.636318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.636264 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"default-dockercfg-jmw9g\"" Oct 31 12:38:20.637400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.637378 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"openshift-service-ca.crt\"" Oct 31 12:38:20.638809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.638789 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb"] Oct 31 12:38:20.742640 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.742602 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8q2\" (UniqueName: \"kubernetes.io/projected/11cf690d-d2e4-426c-8b02-38a801dff8a7-kube-api-access-fz8q2\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.742841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.742649 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cf690d-d2e4-426c-8b02-38a801dff8a7-tls-certs\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.742841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.742686 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-home\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.742841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.742761 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-model-cache\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.742841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.742797 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-dshm\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.743054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.742930 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-tmp-dir\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.834491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.834437 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:20.843758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.843735 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-dshm\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.843882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.843834 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-tmp-dir\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.843939 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.843891 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8q2\" (UniqueName: \"kubernetes.io/projected/11cf690d-d2e4-426c-8b02-38a801dff8a7-kube-api-access-fz8q2\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.843939 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.843907 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cf690d-d2e4-426c-8b02-38a801dff8a7-tls-certs\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.844047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.844017 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-home\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.844132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.844108 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-model-cache\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.844286 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.844266 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-tmp-dir\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.844357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.844331 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-home\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.844446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.844426 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-model-cache\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.846206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.846180 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-dshm\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.846396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.846382 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cf690d-d2e4-426c-8b02-38a801dff8a7-tls-certs\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.863484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.863446 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8q2\" (UniqueName: \"kubernetes.io/projected/11cf690d-d2e4-426c-8b02-38a801dff8a7-kube-api-access-fz8q2\") pod \"premium-w6-67d410-kserve-7f8c876c7d-hchhb\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:20.941787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:20.941738 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:21.217557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.217534 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb"] Oct 31 12:38:21.219217 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:38:21.219186 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11cf690d_d2e4_426c_8b02_38a801dff8a7.slice/crio-8ab3b79287780b7aee6d32b961a1ffa1e4107e323087bfaf2af483d5f3c95dc9 WatchSource:0}: Error finding container 8ab3b79287780b7aee6d32b961a1ffa1e4107e323087bfaf2af483d5f3c95dc9: Status 404 returned error can't find the container with id 8ab3b79287780b7aee6d32b961a1ffa1e4107e323087bfaf2af483d5f3c95dc9 Oct 31 12:38:21.237266 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.237249 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:21.859781 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.859381 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz"] Oct 31 12:38:21.864261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.864237 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:21.870759 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.870738 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"kube-root-ca.crt\"" Oct 31 12:38:21.871876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.871851 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"premium-w4-c67de2-kserve-self-signed-certs\"" Oct 31 12:38:21.872021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.871860 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"openshift-service-ca.crt\"" Oct 31 12:38:21.872121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.871912 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"default-dockercfg-n9qq6\"" Oct 31 12:38:21.875212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.875186 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz"] Oct 31 12:38:21.926871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.926840 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" event={"ID":"11cf690d-d2e4-426c-8b02-38a801dff8a7","Type":"ContainerStarted","Data":"159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40"} Oct 31 12:38:21.926871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.926870 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" event={"ID":"11cf690d-d2e4-426c-8b02-38a801dff8a7","Type":"ContainerStarted","Data":"8ab3b79287780b7aee6d32b961a1ffa1e4107e323087bfaf2af483d5f3c95dc9"} Oct 31 12:38:21.935996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.935970 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:21.942855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.942836 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:21.946249 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.946210 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podStartSLOduration=1.946199257 podStartE2EDuration="1.946199257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:21.945555678 +0000 UTC m=+612.760570071" watchObservedRunningTime="2026-10-31 12:38:21.946199257 +0000 UTC m=+612.761213666" Oct 31 12:38:21.956097 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.956075 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdw85\" (UniqueName: \"kubernetes.io/projected/a0348efe-1a63-4ab5-9163-61a781568c31-kube-api-access-bdw85\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:21.956188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.956105 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-model-cache\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:21.956188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.956128 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-home\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:21.956328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.956307 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-dshm\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:21.956388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.956358 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0348efe-1a63-4ab5-9163-61a781568c31-tls-certs\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:21.956553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:21.956530 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-tmp-dir\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.058114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.058008 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-home\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.058442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.058406 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-home\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.058581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.058558 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-dshm\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.058670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.058598 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0348efe-1a63-4ab5-9163-61a781568c31-tls-certs\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.058710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.058673 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-tmp-dir\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.058816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.058796 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdw85\" (UniqueName: \"kubernetes.io/projected/a0348efe-1a63-4ab5-9163-61a781568c31-kube-api-access-bdw85\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.058975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.058957 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-model-cache\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.059563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.059532 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-model-cache\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.059818 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.059800 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-tmp-dir\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.081560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.064040 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:22.081560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.064490 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0348efe-1a63-4ab5-9163-61a781568c31-tls-certs\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.081560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.081423 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-dshm\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.090508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.090454 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdw85\" (UniqueName: \"kubernetes.io/projected/a0348efe-1a63-4ab5-9163-61a781568c31-kube-api-access-bdw85\") pod \"premium-w4-c67de2-kserve-6fdd7d5557-w9gjz\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.181921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.181851 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:22.461140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.461112 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz"] Oct 31 12:38:22.462711 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:38:22.462679 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0348efe_1a63_4ab5_9163_61a781568c31.slice/crio-931a2662a373e6a71ca2a16f1d63037a23cf4dc66faba9d268cb667bc9afff1c WatchSource:0}: Error finding container 931a2662a373e6a71ca2a16f1d63037a23cf4dc66faba9d268cb667bc9afff1c: Status 404 returned error can't find the container with id 931a2662a373e6a71ca2a16f1d63037a23cf4dc66faba9d268cb667bc9afff1c Oct 31 12:38:22.639648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.639579 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:22.927102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.927065 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:22.933002 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.932976 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" event={"ID":"a0348efe-1a63-4ab5-9163-61a781568c31","Type":"ContainerStarted","Data":"fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e"} Oct 31 12:38:22.933136 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.933013 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" event={"ID":"a0348efe-1a63-4ab5-9163-61a781568c31","Type":"ContainerStarted","Data":"931a2662a373e6a71ca2a16f1d63037a23cf4dc66faba9d268cb667bc9afff1c"} Oct 31 12:38:22.953637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:22.953596 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podStartSLOduration=1.9535826250000001 podStartE2EDuration="1.953582625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:22.951635389 +0000 UTC m=+613.766649854" watchObservedRunningTime="2026-10-31 12:38:22.953582625 +0000 UTC m=+613.768597083" Oct 31 12:38:23.182864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:23.182785 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:23.415851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:23.415819 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:23.546342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:23.546251 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:23.558614 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:23.558589 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:23.948806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:23.948781 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:38:24.231869 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:24.231794 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:24.786521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:24.786483 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:38:24.941798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:24.941765 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:25.704334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:25.704304 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:25.831411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:25.831374 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:26.550404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:26.550361 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:27.300310 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:27.300273 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:28.030289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:28.030252 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:28.724779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:28.724752 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:29.421610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:29.421566 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:30.239762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:30.239725 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:30.342428 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:38:30.342393 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:38:30.942531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:30.942501 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:30.959081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:30.959052 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:30.999971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:30.999948 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:38:31.001386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:31.001362 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:31.738585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:31.738552 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:32.182724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:32.182690 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:32.194846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:32.194823 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:32.423022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:32.422996 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:33.010212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:33.010185 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:38:33.116095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:33.116060 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:33.831426 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:33.831397 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:33.975888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:33.975848 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:34.481722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:34.481686 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:35.402152 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:35.402113 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:36.136498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:36.136445 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:36.934751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:36.934709 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:37.681123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:37.681090 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:37.828022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:37.827990 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:38.524089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:38.524058 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:39.295437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:39.295406 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:40.045035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:40.045000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:40.724327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:40.724300 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:41.414599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:41.414571 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:42.088441 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:42.088405 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:42.831385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:42.830830 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:43.479841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:43.479796 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:44.141611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:44.141583 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:45.173943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:45.173906 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:46.192124 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:46.192094 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:46.553445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:46.553374 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:38:46.851694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:46.851663 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:47.862298 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:47.862258 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:48.628897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:48.628864 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:49.427547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:49.427515 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:50.265328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:50.265273 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:51.121492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.121431 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:51.603438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.603409 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7"] Oct 31 12:38:51.608853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.608823 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.617957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.617929 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fb0343-openshift-default-dockercfg-pktb6\"" Oct 31 12:38:51.621491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.621440 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7"] Oct 31 12:38:51.710355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710319 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710372 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710400 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710487 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710526 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chx4p\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-kube-api-access-chx4p\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710587 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710623 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710739 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.710834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.710778 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812074 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812110 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812128 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812237 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812276 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chx4p\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-kube-api-access-chx4p\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812320 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812344 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812398 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812428 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812519 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812744 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812849 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812913 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.812979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.812927 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.814695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.814674 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.814695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.814690 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.824202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.824183 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chx4p\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-kube-api-access-chx4p\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.824309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.824236 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:51.837728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.837708 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:51.923525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:51.923428 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:52.208766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:52.208744 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7"] Oct 31 12:38:52.210554 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:38:52.210523 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc8ebb8_d3bf_4c04_a793_25e2810b70ef.slice/crio-d644c21a4644a0f5fde096ade07ec7caee3ebea6dd8bf17a18ee5806b7580b70 WatchSource:0}: Error finding container d644c21a4644a0f5fde096ade07ec7caee3ebea6dd8bf17a18ee5806b7580b70: Status 404 returned error can't find the container with id d644c21a4644a0f5fde096ade07ec7caee3ebea6dd8bf17a18ee5806b7580b70 Oct 31 12:38:52.213019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:52.212986 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:38:52.213119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:52.213061 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:38:52.213119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:52.213091 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:38:52.637845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:52.637815 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:53.116647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:53.116615 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" event={"ID":"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef","Type":"ContainerStarted","Data":"fad0c3791b26a357167106109b39ce240a3db6f2da4fa72e0058d7dd0aa948f6"} Oct 31 12:38:53.116829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:53.116669 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" event={"ID":"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef","Type":"ContainerStarted","Data":"d644c21a4644a0f5fde096ade07ec7caee3ebea6dd8bf17a18ee5806b7580b70"} Oct 31 12:38:53.425971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:53.425898 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:53.924442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:53.924411 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:53.925899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:53.925873 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:38:53.926071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:53.925918 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 12:38:54.199746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:54.199645 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:54.924218 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:54.924185 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:38:54.924655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:54.924235 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 12:38:54.944194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:54.944168 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:55.749899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:55.749865 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:55.924617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:55.924587 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:38:55.924993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:55.924639 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 12:38:56.863478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:56.863363 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:56.942611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:56.942573 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:38:56.943232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:56.943196 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 12:38:57.789141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:57.789098 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:57.924699 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:57.924665 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:38:57.924863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:57.924713 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 12:38:58.660490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:58.660443 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:59.387973 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:59.387935 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:59.502521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:59.502492 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:59.502911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:59.502892 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:59.504263 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:59.504237 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:38:59.525867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:38:59.525807 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podStartSLOduration=8.525795121 podStartE2EDuration="8.525795121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:53.143409953 +0000 UTC m=+643.958424343" watchObservedRunningTime="2026-10-31 12:38:59.525795121 +0000 UTC m=+650.340809511" Oct 31 12:39:00.139217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:00.139190 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:00.340982 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:39:00.340948 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:39:01.129924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:01.129896 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:01.948139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:01.948109 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:02.836668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:02.836638 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:03.541397 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:03.541362 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:04.237682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:04.237655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:05.023652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:05.023625 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:05.733130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:05.733100 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:06.443792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:06.443763 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:06.582582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:06.582554 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:06.998571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:06.998540 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb"] Oct 31 12:39:07.002852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.002826 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.009261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.009238 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b218d3-openshift-default-dockercfg-kz5nw\"" Oct 31 12:39:07.014304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.014280 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb"] Oct 31 12:39:07.069957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.069930 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.069970 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.069992 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q78np\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-kube-api-access-q78np\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.070133 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.070166 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.070194 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.070251 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771d25b0-f843-4e87-8424-cfbb691f3173-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.070283 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771d25b0-f843-4e87-8424-cfbb691f3173-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.070427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.070347 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.171641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.171604 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.171859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.171668 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.171859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.171719 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.171859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.171741 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771d25b0-f843-4e87-8424-cfbb691f3173-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.171859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.171762 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771d25b0-f843-4e87-8424-cfbb691f3173-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.171859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.171828 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.172075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.171962 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.172075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.172015 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.172075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.172055 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q78np\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-kube-api-access-q78np\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.172929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.172888 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.173547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.173522 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.174185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.174160 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.174781 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.174735 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771d25b0-f843-4e87-8424-cfbb691f3173-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.174970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.174948 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.176337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.176317 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.176337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.176326 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771d25b0-f843-4e87-8424-cfbb691f3173-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.180895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.180872 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:07.187667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.185653 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.187667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.186438 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q78np\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-kube-api-access-q78np\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.256943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.256884 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw"] Oct 31 12:39:07.261863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.261844 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.275070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.275047 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-85ce08-openshift-default-dockercfg-5nn7z\"" Oct 31 12:39:07.275444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.275429 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw"] Oct 31 12:39:07.317787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.317767 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:07.374618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374371 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.374618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374501 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.374618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374597 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.374827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374631 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.374827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374657 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.374827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374703 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.374827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374727 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.375030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374826 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zch\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-kube-api-access-74zch\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.375030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.374899 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.475698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.475663 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.475781 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.475829 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476057 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476105 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476132 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476137 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476178 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476205 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476273 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zch\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-kube-api-access-74zch\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476415 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476434 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476525 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.476726 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.476705 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.478869 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.478847 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.479048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.479031 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.487790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.487770 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.487867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.487824 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zch\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-kube-api-access-74zch\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.576874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.576854 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:07.596429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.596409 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb"] Oct 31 12:39:07.598348 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:39:07.598321 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771d25b0_f843_4e87_8424_cfbb691f3173.slice/crio-647fbc9844ecbdf39bfa950c3b86ecdb15e2bec48ef596f93226cca5ba69bdae WatchSource:0}: Error finding container 647fbc9844ecbdf39bfa950c3b86ecdb15e2bec48ef596f93226cca5ba69bdae: Status 404 returned error can't find the container with id 647fbc9844ecbdf39bfa950c3b86ecdb15e2bec48ef596f93226cca5ba69bdae Oct 31 12:39:07.601392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.601340 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:39:07.601486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.601442 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:39:07.601534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.601518 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:39:07.861452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.861426 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw"] Oct 31 12:39:07.863308 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:39:07.863280 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9edd6c1_cabc_48e4_98c0_78d0d012af06.slice/crio-c43515593a9d42eef2a0aa6534c69d3c8d8de2e7a8b7a097ac4b15c50093895a WatchSource:0}: Error finding container c43515593a9d42eef2a0aa6534c69d3c8d8de2e7a8b7a097ac4b15c50093895a: Status 404 returned error can't find the container with id c43515593a9d42eef2a0aa6534c69d3c8d8de2e7a8b7a097ac4b15c50093895a Oct 31 12:39:07.865757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.865726 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:39:07.865824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.865790 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:39:07.865824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.865817 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:39:07.935201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:07.935175 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:08.214028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.213356 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" event={"ID":"d9edd6c1-cabc-48e4-98c0-78d0d012af06","Type":"ContainerStarted","Data":"630d80cb32b11580807edcca86f400263dddbda3907daa79b926e3e3a61b3d5e"} Oct 31 12:39:08.214028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.213423 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" event={"ID":"d9edd6c1-cabc-48e4-98c0-78d0d012af06","Type":"ContainerStarted","Data":"c43515593a9d42eef2a0aa6534c69d3c8d8de2e7a8b7a097ac4b15c50093895a"} Oct 31 12:39:08.215480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.215439 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" event={"ID":"771d25b0-f843-4e87-8424-cfbb691f3173","Type":"ContainerStarted","Data":"6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09"} Oct 31 12:39:08.215592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.215496 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" event={"ID":"771d25b0-f843-4e87-8424-cfbb691f3173","Type":"ContainerStarted","Data":"647fbc9844ecbdf39bfa950c3b86ecdb15e2bec48ef596f93226cca5ba69bdae"} Oct 31 12:39:08.238149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.238111 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podStartSLOduration=1.238097349 podStartE2EDuration="1.238097349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:08.23517343 +0000 UTC m=+659.050187822" watchObservedRunningTime="2026-10-31 12:39:08.238097349 +0000 UTC m=+659.053111740" Oct 31 12:39:08.258978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.258932 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podStartSLOduration=2.258919937 podStartE2EDuration="2.258919937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:08.25768537 +0000 UTC m=+659.072699764" watchObservedRunningTime="2026-10-31 12:39:08.258919937 +0000 UTC m=+659.073934328" Oct 31 12:39:08.318934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.318904 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:08.320008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.319987 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 12:39:08.320102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.320043 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 12:39:08.583483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.580525 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:08.595492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.584568 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 12:39:08.595492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.584632 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 12:39:08.880518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:08.879828 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:09.318933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:09.318851 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 12:39:09.318933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:09.318908 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 12:39:09.594490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:09.593778 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 12:39:09.594490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:09.593836 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 12:39:09.815682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:09.815414 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:10.318881 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:10.318839 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 12:39:10.319041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:10.318909 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 12:39:10.581884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:10.581852 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 12:39:10.582154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:10.582125 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 12:39:10.732741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:10.732706 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:11.318985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:11.318935 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 12:39:11.319188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:11.319001 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 12:39:11.589250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:11.588813 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 12:39:11.589250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:11.588871 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 12:39:11.631098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:11.631066 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:12.318853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:12.318811 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 12:39:12.319301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:12.318878 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 12:39:12.584515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:12.581790 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 12:39:12.584515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:12.581850 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 12:39:12.607447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:12.607304 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:13.318770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:13.318735 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 12:39:13.318966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:13.318795 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 12:39:13.497425 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:13.497388 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:13.579484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:13.578776 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 12:39:13.579484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:13.578838 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 12:39:14.231015 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.230975 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:14.243103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.242839 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:39:14.324400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.324370 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:14.324867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.324692 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:14.325635 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.325612 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:39:14.581559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.581531 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:14.581923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.581903 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:14.582687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:14.582666 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:39:15.074445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:15.074418 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:15.732632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:15.732600 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:16.441169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:16.441130 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:17.125280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:17.125249 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:17.842930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:17.842867 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:18.698761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:18.698722 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:19.423348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:19.423321 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:20.145520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:20.145485 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:20.820777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:20.820723 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:21.555441 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:21.555408 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:22.246638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:22.246602 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:23.039801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:23.039767 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:23.859483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:23.859015 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:24.627687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:24.627655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:25.332902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:25.332867 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:26.044755 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:26.044713 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:26.789653 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:26.789601 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:27.415865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:27.415828 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:28.121953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:28.121923 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:28.835951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:28.835881 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:29.535878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:29.535847 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:30.341428 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:39:30.341398 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:39:30.504127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:30.504093 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:31.333319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:31.333280 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:32.040989 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:32.040960 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:32.773822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:32.773792 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:33.445151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:33.445123 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:34.137285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:34.137256 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:34.838173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:34.838142 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:35.540370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:35.540336 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:36.216478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:36.216436 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:36.927773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:36.927740 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:37.717746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:37.717715 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:38.443949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:38.443919 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:39.932337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:39.932234 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:40.792333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:40.792299 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:41.709380 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:41.709343 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:42.524612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:42.524583 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:43.241390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:43.241355 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:43.959222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:43.959190 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:44.825685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:44.825649 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:45.371754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:45.371719 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:45.629606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:45.629527 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:46.344117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:46.344083 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:47.047212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:47.047178 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:47.750672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:47.750637 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:48.431936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:48.431906 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:49.343407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:49.343375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:50.109961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:50.109925 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:50.373804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:50.373735 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:39:51.128821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:51.128787 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:52.821638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:52.821591 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 31 12:39:52.991170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:52.990193 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:39:53.011266 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:53.010402 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:39:53.187883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:53.187535 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:39:53.908486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:53.908344 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:54.143497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:54.143314 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:54.150678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:54.150621 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:39:54.150839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:54.150697 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:39:55.337725 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:55.337677 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:56.245882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:56.245847 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:57.185224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:57.185182 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:57.946875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:57.946837 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:58.674724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:58.674683 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:59.548506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.548131 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:59.846129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.846098 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn"] Oct 31 12:39:59.851285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.851265 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:39:59.857303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.857280 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"openshift-service-ca.crt\"" Oct 31 12:39:59.858151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.858120 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"kube-root-ca.crt\"" Oct 31 12:39:59.860633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.860609 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"default-dockercfg-tqnrh\"" Oct 31 12:39:59.860738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.860641 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"distinct-2-w3-b218d3-kserve-self-signed-certs\"" Oct 31 12:39:59.862005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.861978 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn"] Oct 31 12:39:59.924845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:39:59.924819 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:00.001928 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.001873 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-dshm\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.002157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.001932 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-tmp-dir\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.002157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.002073 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-home\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.002157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.002122 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-model-cache\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.002337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.002296 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6h49\" (UniqueName: \"kubernetes.io/projected/06585f3d-576a-4abd-94b5-1ae90777820b-kube-api-access-d6h49\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.002391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.002339 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06585f3d-576a-4abd-94b5-1ae90777820b-tls-certs\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.103708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.103616 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6h49\" (UniqueName: \"kubernetes.io/projected/06585f3d-576a-4abd-94b5-1ae90777820b-kube-api-access-d6h49\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.103708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.103667 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06585f3d-576a-4abd-94b5-1ae90777820b-tls-certs\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.103957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.103754 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-dshm\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.103957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.103778 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-tmp-dir\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.103957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.103856 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-home\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.103957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.103894 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-model-cache\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.104303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.104274 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-home\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.104398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.104301 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-model-cache\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.104398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.104341 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-tmp-dir\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.106173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.106154 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-dshm\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.106475 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.106442 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06585f3d-576a-4abd-94b5-1ae90777820b-tls-certs\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.115707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.115681 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6h49\" (UniqueName: \"kubernetes.io/projected/06585f3d-576a-4abd-94b5-1ae90777820b-kube-api-access-d6h49\") pod \"distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.166433 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.166406 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:00.341720 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:40:00.341686 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:40:00.349069 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.349040 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:00.462316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.462291 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn"] Oct 31 12:40:00.463871 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:00.463840 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06585f3d_576a_4abd_94b5_1ae90777820b.slice/crio-6a49cae092164dec4cfed51f7f3e3b147dd4a7a385d4816e9c2f3e0dc6c3ff51 WatchSource:0}: Error finding container 6a49cae092164dec4cfed51f7f3e3b147dd4a7a385d4816e9c2f3e0dc6c3ff51: Status 404 returned error can't find the container with id 6a49cae092164dec4cfed51f7f3e3b147dd4a7a385d4816e9c2f3e0dc6c3ff51 Oct 31 12:40:00.884000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.883957 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" event={"ID":"06585f3d-576a-4abd-94b5-1ae90777820b","Type":"ContainerStarted","Data":"09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71"} Oct 31 12:40:00.884379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.884013 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" event={"ID":"06585f3d-576a-4abd-94b5-1ae90777820b","Type":"ContainerStarted","Data":"6a49cae092164dec4cfed51f7f3e3b147dd4a7a385d4816e9c2f3e0dc6c3ff51"} Oct 31 12:40:00.906009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:00.905953 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podStartSLOduration=1.905938952 podStartE2EDuration="1.905938952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:00.904735397 +0000 UTC m=+711.719749786" watchObservedRunningTime="2026-10-31 12:40:00.905938952 +0000 UTC m=+711.720953321" Oct 31 12:40:01.125931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:01.125903 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:01.167148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:01.167065 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:01.952934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:01.952904 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:02.887913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:02.887880 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:03.841656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:03.841616 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:04.608655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:04.608629 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:05.456922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:05.456862 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:06.421618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:06.421587 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:06.615137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:06.615105 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:07.144148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.144117 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c"] Oct 31 12:40:07.148306 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.148281 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.167856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.167832 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c"] Oct 31 12:40:07.281099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281054 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281137 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281249 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281288 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvshm\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-kube-api-access-bvshm\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281320 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281366 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281419 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281576 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89263210-c66c-4aea-8a82-0849d93f0a7a-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.281694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.281647 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.382946 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.382903 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.382971 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383001 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvshm\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-kube-api-access-bvshm\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383042 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383096 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383160 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383205 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89263210-c66c-4aea-8a82-0849d93f0a7a-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383267 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383324 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383608 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383661 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.383966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.383906 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.384275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.384247 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89263210-c66c-4aea-8a82-0849d93f0a7a-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.384603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.384580 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.386009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.385976 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.386208 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.386184 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.403667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.403582 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvshm\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-kube-api-access-bvshm\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.404421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.404325 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.464486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.464444 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:07.481715 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.481687 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:07.773196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.773169 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c"] Oct 31 12:40:07.774299 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:07.774273 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89263210_c66c_4aea_8a82_0849d93f0a7a.slice/crio-93fc917a90cfa0c4d2f52363584bc84727b7b5f5e6dd2daabf6bb4d024f35b61 WatchSource:0}: Error finding container 93fc917a90cfa0c4d2f52363584bc84727b7b5f5e6dd2daabf6bb4d024f35b61: Status 404 returned error can't find the container with id 93fc917a90cfa0c4d2f52363584bc84727b7b5f5e6dd2daabf6bb4d024f35b61 Oct 31 12:40:07.776619 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.776589 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:07.776682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.776651 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:07.776720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.776678 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:07.932317 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.932283 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" event={"ID":"89263210-c66c-4aea-8a82-0849d93f0a7a","Type":"ContainerStarted","Data":"5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e"} Oct 31 12:40:07.932518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.932335 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" event={"ID":"89263210-c66c-4aea-8a82-0849d93f0a7a","Type":"ContainerStarted","Data":"93fc917a90cfa0c4d2f52363584bc84727b7b5f5e6dd2daabf6bb4d024f35b61"} Oct 31 12:40:07.975757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:07.975701 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podStartSLOduration=0.975683564 podStartE2EDuration="975.683564ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:07.972266475 +0000 UTC m=+718.787280869" watchObservedRunningTime="2026-10-31 12:40:07.975683564 +0000 UTC m=+718.790697955" Oct 31 12:40:08.410927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:08.410884 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:08.486780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:08.484851 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:08.487602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:08.487570 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:08.487842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:08.487815 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:09.479558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:09.467773 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:09.479558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:09.467828 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:09.603411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:09.603378 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:10.178488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:10.178253 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:10.472540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:10.471934 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:10.472540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:10.471998 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:10.947496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:10.947428 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:40:10.999486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:10.996857 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:40:11.015731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.015693 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:40:11.181621 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.181574 6727 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:40:11.497490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.495903 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:11.497490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.495972 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:11.886487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.885560 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:40:11.947228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.947173 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:40:11.948807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.947776 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:40:11.948807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:11.948089 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:40:12.152207 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:12.150553 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:40:12.274324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:12.274292 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:12.494539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:12.487937 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:12.494539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:12.487993 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:13.433597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.433566 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:13.482966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.467433 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:13.482966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.467510 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:13.588420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.588381 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq"] Oct 31 12:40:13.603054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.601045 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.607792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.607653 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq"] Oct 31 12:40:13.609098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.609075 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"distinct-w0-fb0343-kserve-self-signed-certs\"" Oct 31 12:40:13.645073 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.645048 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-home\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.645201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.645111 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzcg\" (UniqueName: \"kubernetes.io/projected/543a8bb9-7cd3-4055-8109-7fb9d60590a8-kube-api-access-wqzcg\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.645261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.645205 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-dshm\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.645298 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.645267 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tmp-dir\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.645494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.645449 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tls-certs\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.645580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.645502 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-model-cache\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.692608 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.692535 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:13.746745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.746720 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tls-certs\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.746867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.746765 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-model-cache\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.746867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.746818 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-home\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.746990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.746892 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzcg\" (UniqueName: \"kubernetes.io/projected/543a8bb9-7cd3-4055-8109-7fb9d60590a8-kube-api-access-wqzcg\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.746990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.746932 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-dshm\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.746990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.746970 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tmp-dir\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.747330 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.747302 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-model-cache\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.747406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.747349 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tmp-dir\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.747406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.747369 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-home\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.749350 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.749330 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tls-certs\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.749513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.749489 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-dshm\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.774780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.774707 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzcg\" (UniqueName: \"kubernetes.io/projected/543a8bb9-7cd3-4055-8109-7fb9d60590a8-kube-api-access-wqzcg\") pod \"distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:13.918727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:13.918696 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:14.225772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.225747 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq"] Oct 31 12:40:14.227508 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:14.227458 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod543a8bb9_7cd3_4055_8109_7fb9d60590a8.slice/crio-a97b9166fd29140f7b28666ab5c15fe1c0a196b3a0e32d7857c6425627b9af3b WatchSource:0}: Error finding container a97b9166fd29140f7b28666ab5c15fe1c0a196b3a0e32d7857c6425627b9af3b: Status 404 returned error can't find the container with id a97b9166fd29140f7b28666ab5c15fe1c0a196b3a0e32d7857c6425627b9af3b Oct 31 12:40:14.382571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.382504 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:14.506100 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.506063 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:14.506654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.506623 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:14.562326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.562146 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs"] Oct 31 12:40:14.602620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.600858 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs"] Oct 31 12:40:14.602620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.600980 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.658778 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.658750 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095de43a-2809-4ebe-aaf1-208d47b55993-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.658934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.658810 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.658934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.658841 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095de43a-2809-4ebe-aaf1-208d47b55993-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.658934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.658896 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.659074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.659028 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.659074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.659055 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6l4l\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-kube-api-access-w6l4l\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.659288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.659266 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.659320 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.659306 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.659401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.659386 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760327 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760379 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095de43a-2809-4ebe-aaf1-208d47b55993-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760419 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760475 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760500 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6l4l\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-kube-api-access-w6l4l\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760652 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760680 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760740 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760753 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760778 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.760897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760810 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095de43a-2809-4ebe-aaf1-208d47b55993-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.761055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.760979 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.761108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.761068 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.761398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.761382 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095de43a-2809-4ebe-aaf1-208d47b55993-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.765531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.765500 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.766753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.766735 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095de43a-2809-4ebe-aaf1-208d47b55993-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.772686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.772665 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6l4l\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-kube-api-access-w6l4l\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.773061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.773040 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-897xs\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:14.918216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:14.918185 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:15.107729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.107692 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" event={"ID":"543a8bb9-7cd3-4055-8109-7fb9d60590a8","Type":"ContainerStarted","Data":"f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7"} Oct 31 12:40:15.107900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.107738 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" event={"ID":"543a8bb9-7cd3-4055-8109-7fb9d60590a8","Type":"ContainerStarted","Data":"a97b9166fd29140f7b28666ab5c15fe1c0a196b3a0e32d7857c6425627b9af3b"} Oct 31 12:40:15.128293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.128241 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podStartSLOduration=2.128228639 podStartE2EDuration="2.128228639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:15.126259554 +0000 UTC m=+725.941273947" watchObservedRunningTime="2026-10-31 12:40:15.128228639 +0000 UTC m=+725.943243028" Oct 31 12:40:15.220260 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:15.220234 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095de43a_2809_4ebe_aaf1_208d47b55993.slice/crio-dcfd62cfda346b9c87adbeaf440981df4ee81620862f50b8e6868be491d791ac WatchSource:0}: Error finding container dcfd62cfda346b9c87adbeaf440981df4ee81620862f50b8e6868be491d791ac: Status 404 returned error can't find the container with id dcfd62cfda346b9c87adbeaf440981df4ee81620862f50b8e6868be491d791ac Oct 31 12:40:15.222651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.222617 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:15.222773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.222699 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:15.222773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.222746 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:15.228475 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.228425 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs"] Oct 31 12:40:15.272191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.272162 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:15.464855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.464818 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 12:40:15.465054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.464891 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 12:40:15.465655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.465632 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:15.919487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:15.919429 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:16.115669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:16.115610 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" event={"ID":"095de43a-2809-4ebe-aaf1-208d47b55993","Type":"ContainerStarted","Data":"280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6"} Oct 31 12:40:16.115875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:16.115674 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" event={"ID":"095de43a-2809-4ebe-aaf1-208d47b55993","Type":"ContainerStarted","Data":"dcfd62cfda346b9c87adbeaf440981df4ee81620862f50b8e6868be491d791ac"} Oct 31 12:40:16.139555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:16.139505 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podStartSLOduration=2.139493594 podStartE2EDuration="2.139493594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:16.138077092 +0000 UTC m=+726.953091489" watchObservedRunningTime="2026-10-31 12:40:16.139493594 +0000 UTC m=+726.954507977" Oct 31 12:40:16.276544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:16.274950 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:16.921484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:16.918998 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:16.921484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:16.920930 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:40:16.921484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:16.920969 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:40:17.095488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:17.094757 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:17.233594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:17.233510 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:17.233938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:17.233919 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:17.234622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:17.234600 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:40:17.918822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:17.918786 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:40:17.918991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:17.918875 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:40:18.202406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:18.202297 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:18.919422 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:18.919392 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:40:18.919576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:18.919442 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:40:19.159176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:19.159130 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:19.919388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:19.919302 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:40:19.919824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:19.919379 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:40:20.007182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:20.007142 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:20.198644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:20.198570 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:20.209704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:20.209683 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:40:20.761312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:20.761281 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:20.918686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:20.918648 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:40:20.918888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:20.918710 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:40:21.544193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.544154 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:21.745313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.745273 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn"] Oct 31 12:40:21.749421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.749399 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.756019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.755992 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"distinct-w1-85ce08-kserve-self-signed-certs\"" Oct 31 12:40:21.758531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.758513 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"default-dockercfg-t7qtc\"" Oct 31 12:40:21.762409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.762385 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn"] Oct 31 12:40:21.814861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.814802 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"kube-root-ca.crt\"" Oct 31 12:40:21.814861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.814827 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"openshift-service-ca.crt\"" Oct 31 12:40:21.821162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.821141 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:21.894941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.894907 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69da0e2c-7478-499f-879e-20aa90d6ad89-tls-certs\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.895128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.894990 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-home\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.895128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.895062 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-tmp-dir\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.895128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.895087 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-dshm\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.895308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.895167 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vgkf\" (UniqueName: \"kubernetes.io/projected/69da0e2c-7478-499f-879e-20aa90d6ad89-kube-api-access-5vgkf\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.895363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.895310 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-model-cache\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.922851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.922816 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:40:21.923006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.922877 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:40:21.996269 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996237 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-home\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.996437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996312 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-tmp-dir\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.996437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996336 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-dshm\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.996437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996407 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vgkf\" (UniqueName: \"kubernetes.io/projected/69da0e2c-7478-499f-879e-20aa90d6ad89-kube-api-access-5vgkf\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.996655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996626 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-model-cache\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.996738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996725 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69da0e2c-7478-499f-879e-20aa90d6ad89-tls-certs\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.996799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996753 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-tmp-dir\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.996799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.996766 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-home\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.997145 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.997126 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-model-cache\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.998833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.998815 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-dshm\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:21.999658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:21.999627 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69da0e2c-7478-499f-879e-20aa90d6ad89-tls-certs\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:22.011974 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.011954 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vgkf\" (UniqueName: \"kubernetes.io/projected/69da0e2c-7478-499f-879e-20aa90d6ad89-kube-api-access-5vgkf\") pod \"distinct-w1-85ce08-kserve-6d7cf8c559-52cqn\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:22.062848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.062816 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:22.285365 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.285331 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8"] Oct 31 12:40:22.290438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.290418 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.314111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.313955 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8"] Oct 31 12:40:22.352524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.352502 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn"] Oct 31 12:40:22.356150 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:22.356120 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69da0e2c_7478_499f_879e_20aa90d6ad89.slice/crio-76a13a736411306695284a2279b44624428c1c56d4ffdd059cfd0b501ae53b78 WatchSource:0}: Error finding container 76a13a736411306695284a2279b44624428c1c56d4ffdd059cfd0b501ae53b78: Status 404 returned error can't find the container with id 76a13a736411306695284a2279b44624428c1c56d4ffdd059cfd0b501ae53b78 Oct 31 12:40:22.413508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.413479 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.413613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.413576 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.413707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.413688 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.413910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.413889 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.413970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.413945 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vggn\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-kube-api-access-9vggn\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.414074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.414054 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.414143 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.414127 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.414212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.414191 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.414273 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.414263 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.492701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.492661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:22.515653 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515616 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.515839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515661 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.515839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515693 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.515839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515776 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.515839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515807 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515878 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515934 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.515999 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vggn\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-kube-api-access-9vggn\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.516060 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.516067 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.516096 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.516361 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.516542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.516522 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.517139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.517113 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.518899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.518872 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.519362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.519336 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.539601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.539580 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vggn\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-kube-api-access-9vggn\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.540071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.540054 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.619006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.618959 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:22.910691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.910658 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8"] Oct 31 12:40:22.912122 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:22.912090 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bff0de6_37e4_4e89_b0e8_fd7a153c0098.slice/crio-82db206ca5a2dff1d05f03b95bc45ac3e617a0cc6c0ce102662587625ed9d8a9 WatchSource:0}: Error finding container 82db206ca5a2dff1d05f03b95bc45ac3e617a0cc6c0ce102662587625ed9d8a9: Status 404 returned error can't find the container with id 82db206ca5a2dff1d05f03b95bc45ac3e617a0cc6c0ce102662587625ed9d8a9 Oct 31 12:40:22.915061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.915028 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:22.915141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.915103 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:22.915141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:22.915136 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:23.181576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.180979 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" event={"ID":"6bff0de6-37e4-4e89-b0e8-fd7a153c0098","Type":"ContainerStarted","Data":"46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33"} Oct 31 12:40:23.181576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.181049 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" event={"ID":"6bff0de6-37e4-4e89-b0e8-fd7a153c0098","Type":"ContainerStarted","Data":"82db206ca5a2dff1d05f03b95bc45ac3e617a0cc6c0ce102662587625ed9d8a9"} Oct 31 12:40:23.183844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.183817 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" event={"ID":"69da0e2c-7478-499f-879e-20aa90d6ad89","Type":"ContainerStarted","Data":"97d8c53aab1f2cd7db2c3e271bd5a14acf700daea9443dda8f4011c5dd61b504"} Oct 31 12:40:23.184188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.183857 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" event={"ID":"69da0e2c-7478-499f-879e-20aa90d6ad89","Type":"ContainerStarted","Data":"76a13a736411306695284a2279b44624428c1c56d4ffdd059cfd0b501ae53b78"} Oct 31 12:40:23.208084 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.208030 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podStartSLOduration=1.207998701 podStartE2EDuration="1.207998701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:23.203849823 +0000 UTC m=+734.018864213" watchObservedRunningTime="2026-10-31 12:40:23.207998701 +0000 UTC m=+734.023013091" Oct 31 12:40:23.366669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.366488 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:23.542761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.542019 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:23.620075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.620038 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:23.621648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.621621 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:40:23.621792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.621674 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:40:23.850785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.850755 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:23.851173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.851149 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:23.851859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.851842 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:40:23.886486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.878761 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podStartSLOduration=2.878743588 podStartE2EDuration="2.878743588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:23.224717569 +0000 UTC m=+734.039731961" watchObservedRunningTime="2026-10-31 12:40:23.878743588 +0000 UTC m=+734.693757980" Oct 31 12:40:23.925073 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.919192 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:23.938359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:23.938335 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:24.066592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:24.066534 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:24.204423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:24.204337 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:40:24.305430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:24.305397 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:24.620160 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:24.620131 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:40:24.620554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:24.620176 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:40:25.191146 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:25.191116 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:25.354666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:25.352598 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:25.619748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:25.619711 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:40:25.619926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:25.619786 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:40:26.161756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:26.152655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:26.619541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:26.619517 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:40:26.619756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:26.619735 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:40:26.993833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:26.993689 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:27.620235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:27.620195 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:40:27.620734 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:27.620258 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:40:27.744334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:27.744299 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:28.619404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:28.619371 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:40:28.619665 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:28.619637 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:40:28.653714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:28.653681 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:29.343713 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:29.343662 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:29.623413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:29.623341 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:29.623727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:29.623703 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:29.624369 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:29.624350 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 12:40:30.114355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:30.112881 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:30.265816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:30.265786 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:40:30.344856 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:40:30.344821 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:40:30.923451 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:30.923422 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:32.066575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:32.065669 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:32.087496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:32.085454 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:32.955420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:32.955385 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:33.018126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:33.018099 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:40:33.445595 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:33.445563 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:34.229809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:34.229775 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:35.126396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:35.126367 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:35.400722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:35.400638 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:35.956537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:35.956501 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:36.640662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:36.640634 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:37.445783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:37.445747 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:38.309131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:38.309100 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:39.064006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:39.063969 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:39.863622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:39.863597 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:40.734212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:40.734183 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:41.531438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:41.531406 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:42.246183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:42.246133 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:42.944570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:42.944537 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:43.632068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.632035 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:43.806960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.806928 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8"] Oct 31 12:40:43.812023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.811999 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.820950 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.820911 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"distinct-w6-67d410-kserve-self-signed-certs\"" Oct 31 12:40:43.824766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.824741 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8"] Oct 31 12:40:43.883158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.883101 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-home\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.883276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.883158 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-model-cache\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.883322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.883300 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6tr\" (UniqueName: \"kubernetes.io/projected/176af21f-5986-4784-89c3-9d7b1ab15609-kube-api-access-ts6tr\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.883391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.883376 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-tmp-dir\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.883459 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.883438 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176af21f-5986-4784-89c3-9d7b1ab15609-tls-certs\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.883529 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.883495 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-dshm\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.916626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.916601 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:43.985650 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.985612 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6tr\" (UniqueName: \"kubernetes.io/projected/176af21f-5986-4784-89c3-9d7b1ab15609-kube-api-access-ts6tr\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.986323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.986297 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-tmp-dir\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.987200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.987178 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176af21f-5986-4784-89c3-9d7b1ab15609-tls-certs\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.987359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.987344 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-dshm\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.987510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.987492 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-home\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.987670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.987656 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-model-cache\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.988240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.988219 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-model-cache\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.988365 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.986896 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-tmp-dir\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.991050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.991027 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-home\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.992633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.992613 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-dshm\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:43.993859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:43.993828 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176af21f-5986-4784-89c3-9d7b1ab15609-tls-certs\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:44.007236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:44.006846 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6tr\" (UniqueName: \"kubernetes.io/projected/176af21f-5986-4784-89c3-9d7b1ab15609-kube-api-access-ts6tr\") pod \"distinct-w6-67d410-kserve-65d4d57-9skn8\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:44.127374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:44.127343 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:44.408102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:44.408074 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8"] Oct 31 12:40:44.409726 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:44.409700 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176af21f_5986_4784_89c3_9d7b1ab15609.slice/crio-d43a3df7f58d851f153acf04ead95382b84f53a6a617533704f92c7844546814 WatchSource:0}: Error finding container d43a3df7f58d851f153acf04ead95382b84f53a6a617533704f92c7844546814: Status 404 returned error can't find the container with id d43a3df7f58d851f153acf04ead95382b84f53a6a617533704f92c7844546814 Oct 31 12:40:44.435569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:44.435550 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:45.236258 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:45.236221 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:45.346336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:45.346290 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" event={"ID":"176af21f-5986-4784-89c3-9d7b1ab15609","Type":"ContainerStarted","Data":"983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06"} Oct 31 12:40:45.346587 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:45.346349 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" event={"ID":"176af21f-5986-4784-89c3-9d7b1ab15609","Type":"ContainerStarted","Data":"d43a3df7f58d851f153acf04ead95382b84f53a6a617533704f92c7844546814"} Oct 31 12:40:45.368107 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:45.368048 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podStartSLOduration=2.368030389 podStartE2EDuration="2.368030389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:45.366963792 +0000 UTC m=+756.181978194" watchObservedRunningTime="2026-10-31 12:40:45.368030389 +0000 UTC m=+756.183044780" Oct 31 12:40:46.024948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:46.024919 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:46.127578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:46.127537 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:46.723179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:46.723146 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:47.443270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.443209 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:47.539857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.539823 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv"] Oct 31 12:40:47.544253 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.544235 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.551065 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.551035 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv"] Oct 31 12:40:47.556517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.556490 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"distinct-w4-c67de2-kserve-self-signed-certs\"" Oct 31 12:40:47.617055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.617029 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:47.740202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.740122 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22b2f57-4dc8-4add-9763-16660f67c587-tls-certs\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.740592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.740253 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-home\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.740592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.740358 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-model-cache\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.740592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.740405 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzl9q\" (UniqueName: \"kubernetes.io/projected/a22b2f57-4dc8-4add-9763-16660f67c587-kube-api-access-wzl9q\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.740592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.740451 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-tmp-dir\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.740592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.740533 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-dshm\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.841703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.841668 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-home\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.841884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.841725 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-model-cache\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.841884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.841742 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzl9q\" (UniqueName: \"kubernetes.io/projected/a22b2f57-4dc8-4add-9763-16660f67c587-kube-api-access-wzl9q\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.841884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.841770 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-tmp-dir\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.841884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.841799 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-dshm\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.842141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.841895 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22b2f57-4dc8-4add-9763-16660f67c587-tls-certs\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.842141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.842128 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-home\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.842256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.842208 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-tmp-dir\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.842256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.842220 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-model-cache\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.844092 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.844064 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-dshm\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.844732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.844711 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22b2f57-4dc8-4add-9763-16660f67c587-tls-certs\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.853566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.853536 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzl9q\" (UniqueName: \"kubernetes.io/projected/a22b2f57-4dc8-4add-9763-16660f67c587-kube-api-access-wzl9q\") pod \"distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:47.866432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:47.866400 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:48.151760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:48.151428 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv"] Oct 31 12:40:48.153416 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:48.153389 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22b2f57_4dc8_4add_9763_16660f67c587.slice/crio-82e13db46e250f22ac0e8a3390a82a863b1ef8e3add6c334e9194280dc5f1bdc WatchSource:0}: Error finding container 82e13db46e250f22ac0e8a3390a82a863b1ef8e3add6c334e9194280dc5f1bdc: Status 404 returned error can't find the container with id 82e13db46e250f22ac0e8a3390a82a863b1ef8e3add6c334e9194280dc5f1bdc Oct 31 12:40:48.231336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:48.231313 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:48.366290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:48.366257 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" event={"ID":"a22b2f57-4dc8-4add-9763-16660f67c587","Type":"ContainerStarted","Data":"272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280"} Oct 31 12:40:48.366290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:48.366296 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" event={"ID":"a22b2f57-4dc8-4add-9763-16660f67c587","Type":"ContainerStarted","Data":"82e13db46e250f22ac0e8a3390a82a863b1ef8e3add6c334e9194280dc5f1bdc"} Oct 31 12:40:48.385571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:48.385525 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podStartSLOduration=1.385509865 podStartE2EDuration="1.385509865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:48.384583792 +0000 UTC m=+759.199598183" watchObservedRunningTime="2026-10-31 12:40:48.385509865 +0000 UTC m=+759.200524253" Oct 31 12:40:48.867409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:48.867377 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:49.109777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:49.109744 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:49.937786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:49.937754 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:50.653932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:50.653852 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:51.334523 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.334485 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:51.813669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.813597 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd"] Oct 31 12:40:51.817966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.817941 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.824365 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.824328 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"distinct-2-w1-85ce08-kserve-self-signed-certs\"" Oct 31 12:40:51.827538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.827309 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd"] Oct 31 12:40:51.885671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.885644 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:51.889093 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.889066 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-home\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.889240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.889100 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d09a8c-25dd-4644-8455-0e9c5ebed221-tls-certs\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.889311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.889266 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-dshm\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.889311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.889295 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-tmp-dir\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.889411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.889326 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmqb\" (UniqueName: \"kubernetes.io/projected/54d09a8c-25dd-4644-8455-0e9c5ebed221-kube-api-access-wtmqb\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.889536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.889520 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-model-cache\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.994480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.994326 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-model-cache\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.994669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.994491 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-home\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.994669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.994519 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d09a8c-25dd-4644-8455-0e9c5ebed221-tls-certs\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.994669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.994642 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-dshm\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.994669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.994665 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-tmp-dir\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:51.994914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.994700 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmqb\" (UniqueName: \"kubernetes.io/projected/54d09a8c-25dd-4644-8455-0e9c5ebed221-kube-api-access-wtmqb\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.020482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.995582 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-model-cache\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.020482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:51.995838 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-home\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.022317 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.021426 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-tmp-dir\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.032275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.032242 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmqb\" (UniqueName: \"kubernetes.io/projected/54d09a8c-25dd-4644-8455-0e9c5ebed221-kube-api-access-wtmqb\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.039818 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.039645 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d09a8c-25dd-4644-8455-0e9c5ebed221-tls-certs\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.042951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.042924 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-dshm\") pod \"distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.133166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.133120 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:52.190891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.190860 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:52.428030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.428009 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd"] Oct 31 12:40:52.430363 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:52.430336 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54d09a8c_25dd_4644_8455_0e9c5ebed221.slice/crio-cdb605105b567431c69808f9dfdce9fffce2421cd8481c038d74241f3c08fb9d WatchSource:0}: Error finding container cdb605105b567431c69808f9dfdce9fffce2421cd8481c038d74241f3c08fb9d: Status 404 returned error can't find the container with id cdb605105b567431c69808f9dfdce9fffce2421cd8481c038d74241f3c08fb9d Oct 31 12:40:52.456416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.456391 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7"] Oct 31 12:40:52.461722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.461702 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.476611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.476490 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7"] Oct 31 12:40:52.501346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501321 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22f8t\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-kube-api-access-22f8t\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501378 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501440 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501586 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501628 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0885b27-54d3-4456-8beb-c8be64f65964-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501731 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501764 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501848 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0885b27-54d3-4456-8beb-c8be64f65964-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.501999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.501970 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603516 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603453 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603533 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603564 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0885b27-54d3-4456-8beb-c8be64f65964-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603601 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603625 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22f8t\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-kube-api-access-22f8t\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603656 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603677 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603752 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603796 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0885b27-54d3-4456-8beb-c8be64f65964-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603834 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.603954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.603880 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.604262 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.604228 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.604326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.604241 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.604326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.604289 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0885b27-54d3-4456-8beb-c8be64f65964-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.605975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.605950 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.606314 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.606298 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0885b27-54d3-4456-8beb-c8be64f65964-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.615197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.615175 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.615197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.615180 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22f8t\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-kube-api-access-22f8t\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:52.800735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:52.800674 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:53.088299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.088270 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7"] Oct 31 12:40:53.089676 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:40:53.089627 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0885b27_54d3_4456_8beb_c8be64f65964.slice/crio-62c22e6390db0905c9ffb0e60228cb8f1764dd78255f967b760d241e0aa880c6 WatchSource:0}: Error finding container 62c22e6390db0905c9ffb0e60228cb8f1764dd78255f967b760d241e0aa880c6: Status 404 returned error can't find the container with id 62c22e6390db0905c9ffb0e60228cb8f1764dd78255f967b760d241e0aa880c6 Oct 31 12:40:53.092535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.092496 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:53.092642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.092578 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:53.092642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.092624 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:40:53.162939 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.162908 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:53.413567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.412441 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" event={"ID":"54d09a8c-25dd-4644-8455-0e9c5ebed221","Type":"ContainerStarted","Data":"3fecc22682efbaa15b9238b27e749e8bb7944f82f8c55f7e5df775adb3ab403f"} Oct 31 12:40:53.413567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.412513 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" event={"ID":"54d09a8c-25dd-4644-8455-0e9c5ebed221","Type":"ContainerStarted","Data":"cdb605105b567431c69808f9dfdce9fffce2421cd8481c038d74241f3c08fb9d"} Oct 31 12:40:53.423480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.422059 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" event={"ID":"a0885b27-54d3-4456-8beb-c8be64f65964","Type":"ContainerStarted","Data":"32931eac714d01b04d5d5e9b3b8c04a14bb27717ec3e216e42b6ab91379fe479"} Oct 31 12:40:53.423480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.422125 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" event={"ID":"a0885b27-54d3-4456-8beb-c8be64f65964","Type":"ContainerStarted","Data":"62c22e6390db0905c9ffb0e60228cb8f1764dd78255f967b760d241e0aa880c6"} Oct 31 12:40:53.483492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.478320 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podStartSLOduration=2.478301617 podStartE2EDuration="2.478301617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:53.440168117 +0000 UTC m=+764.255182513" watchObservedRunningTime="2026-10-31 12:40:53.478301617 +0000 UTC m=+764.293316011" Oct 31 12:40:53.483492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.479378 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podStartSLOduration=1.479368395 podStartE2EDuration="1.479368395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:53.476842532 +0000 UTC m=+764.291856922" watchObservedRunningTime="2026-10-31 12:40:53.479368395 +0000 UTC m=+764.294382785" Oct 31 12:40:53.801535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:53.801355 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:40:54.133520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:54.133105 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:40:54.137515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:54.133844 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:40:54.730488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:54.729821 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:54.803907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:54.803569 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:40:54.803907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:54.803633 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:40:55.139221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:55.137591 6727 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:40:55.805488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:55.804897 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:40:55.805488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:55.804972 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:40:56.153729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:56.153701 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:56.801103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:56.801061 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:40:56.801257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:56.801130 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:40:57.038662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:57.038621 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:57.804485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:57.801259 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:40:57.804485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:57.801339 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:40:57.894416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:57.894382 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:57.906934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:57.906909 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:58.558269 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:58.558240 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:40:58.801148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:58.801116 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:40:58.801296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:58.801177 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:40:59.404247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.404217 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8"] Oct 31 12:40:59.413167 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.413148 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8"] Oct 31 12:40:59.413437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.413424 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.418372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.418345 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"default-dockercfg-mscfx\"" Oct 31 12:40:59.419032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.419006 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"distinct-2-w2-e6c79d-kserve-self-signed-certs\"" Oct 31 12:40:59.420372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.420348 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"kube-root-ca.crt\"" Oct 31 12:40:59.420490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.420442 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"openshift-service-ca.crt\"" Oct 31 12:40:59.420798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.420771 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/44cd270f-5484-4e52-83bd-125f6bfa6f8d-kube-api-access-pqd6z\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.420927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.420908 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tmp-dir\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.420969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.420947 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-model-cache\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.421128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.421109 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tls-certs\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.421428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.421407 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-home\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.421603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.421585 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-dshm\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.522878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.522836 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tls-certs\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.522979 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-home\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.523015 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-dshm\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523177 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.523074 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/44cd270f-5484-4e52-83bd-125f6bfa6f8d-kube-api-access-pqd6z\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523177 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.523164 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tmp-dir\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.523186 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-model-cache\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523449 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.523426 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-home\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.523664 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-model-cache\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.523876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.523859 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tmp-dir\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.525318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.525297 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tls-certs\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.525410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.525389 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-dshm\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.534770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.534749 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/44cd270f-5484-4e52-83bd-125f6bfa6f8d-kube-api-access-pqd6z\") pod \"distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.735870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.735791 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:40:59.804495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.804146 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:40:59.804495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:40:59.804191 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:41:00.065256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:00.065232 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8"] Oct 31 12:41:00.067451 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:00.067417 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44cd270f_5484_4e52_83bd_125f6bfa6f8d.slice/crio-9ba5cecbb50a02df163dbf118c915bf007eeafd13997554752b8e33d2e0c44b7 WatchSource:0}: Error finding container 9ba5cecbb50a02df163dbf118c915bf007eeafd13997554752b8e33d2e0c44b7: Status 404 returned error can't find the container with id 9ba5cecbb50a02df163dbf118c915bf007eeafd13997554752b8e33d2e0c44b7 Oct 31 12:41:00.343185 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:41:00.343140 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:41:00.564522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:00.564488 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" event={"ID":"44cd270f-5484-4e52-83bd-125f6bfa6f8d","Type":"ContainerStarted","Data":"6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf"} Oct 31 12:41:00.564699 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:00.564530 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" event={"ID":"44cd270f-5484-4e52-83bd-125f6bfa6f8d","Type":"ContainerStarted","Data":"9ba5cecbb50a02df163dbf118c915bf007eeafd13997554752b8e33d2e0c44b7"} Oct 31 12:41:00.586038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:00.585973 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podStartSLOduration=1.585956398 podStartE2EDuration="1.585956398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:00.584260959 +0000 UTC m=+771.399275352" watchObservedRunningTime="2026-10-31 12:41:00.585956398 +0000 UTC m=+771.400970789" Oct 31 12:41:00.736295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:00.736195 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:41:01.023226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:01.023144 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:41:01.023527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:01.023501 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:41:01.024257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:01.024235 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:41:02.140547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:02.135719 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:41:02.153074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:02.153048 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:41:02.588088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:02.588063 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:41:02.968555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:02.968455 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:03.106961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:03.106924 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:03.860568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:03.860526 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:04.139718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:04.139643 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:41:04.150788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:04.150763 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:41:04.585798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:04.585770 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:05.257546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:05.257514 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:06.160704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:06.160675 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:06.600709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:06.600685 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:06.926288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:06.926198 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:06.982408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:06.982374 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc"] Oct 31 12:41:06.986943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:06.986920 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.000420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.000399 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc"] Oct 31 12:41:07.018976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.018949 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.018988 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.019079 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.019146 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.019214 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smdh\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-kube-api-access-4smdh\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.019236 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.019304 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.019359 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.019456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.019405 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b188d-78b4-495c-937a-9fc6e8150593-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.120912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.120889 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.120921 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.120953 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smdh\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-kube-api-access-4smdh\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121074 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121160 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121133 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121189 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121249 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121230 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121255 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b188d-78b4-495c-937a-9fc6e8150593-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121308 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121405 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121442 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121532 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121550 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.121935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.121916 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b188d-78b4-495c-937a-9fc6e8150593-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.123531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.123510 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.123615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.123562 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.132645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.132616 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smdh\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-kube-api-access-4smdh\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.132936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.132916 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.264678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.264624 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs"] Oct 31 12:41:07.272924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.272902 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.285445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.285415 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs"] Oct 31 12:41:07.302389 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.302365 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:07.324366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324243 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.324366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324325 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.324575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324372 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm79h\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-kube-api-access-jm79h\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.324575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324444 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.324575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324520 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.324764 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324744 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafa41b6-0956-45e8-8b12-481bbbcaa965-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.324936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324796 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.324936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.324823 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.326268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.325148 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.426727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.426679 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm79h\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-kube-api-access-jm79h\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.426863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.426747 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.426863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.426778 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.426951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.426904 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafa41b6-0956-45e8-8b12-481bbbcaa965-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.426951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.426942 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.426959 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427016 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427053 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427131 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427342 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427773 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427797 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427784 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafa41b6-0956-45e8-8b12-481bbbcaa965-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.427918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.427884 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.429821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.429795 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.429932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.429910 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.439394 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.439374 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm79h\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-kube-api-access-jm79h\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.439566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.439545 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.593847 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.593823 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc"] Oct 31 12:41:07.594409 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:07.594390 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6b188d_78b4_495c_937a_9fc6e8150593.slice/crio-3481dd8d992196bd639b08de26ac598a1ef2c64c0395f7b1f40702765188d2a6 WatchSource:0}: Error finding container 3481dd8d992196bd639b08de26ac598a1ef2c64c0395f7b1f40702765188d2a6: Status 404 returned error can't find the container with id 3481dd8d992196bd639b08de26ac598a1ef2c64c0395f7b1f40702765188d2a6 Oct 31 12:41:07.597028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.596995 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:07.597117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.597054 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:07.597117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.597082 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:07.600362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.600340 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:07.610147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.610119 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" event={"ID":"ff6b188d-78b4-495c-937a-9fc6e8150593","Type":"ContainerStarted","Data":"3481dd8d992196bd639b08de26ac598a1ef2c64c0395f7b1f40702765188d2a6"} Oct 31 12:41:07.769454 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.769419 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn"] Oct 31 12:41:07.796600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.796360 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:07.804458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.804426 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn"] Oct 31 12:41:07.804625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.804611 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.893985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.893956 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs"] Oct 31 12:41:07.899403 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:07.898489 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafa41b6_0956_45e8_8b12_481bbbcaa965.slice/crio-e776c9c553801e99c06e56692d2e84defd550d039d34b5ac6a4914bfc785adc0 WatchSource:0}: Error finding container e776c9c553801e99c06e56692d2e84defd550d039d34b5ac6a4914bfc785adc0: Status 404 returned error can't find the container with id e776c9c553801e99c06e56692d2e84defd550d039d34b5ac6a4914bfc785adc0 Oct 31 12:41:07.901592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.901556 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:07.901697 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.901621 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:07.901697 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.901649 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:07.947896 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.947871 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.947927 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.948099 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.948135 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.948166 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.948327 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.948448 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.948590 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:07.948680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:07.948658 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxp89\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-kube-api-access-zxp89\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050073 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050021 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050098 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050139 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050161 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050181 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050246 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050337 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050429 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050481 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxp89\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-kube-api-access-zxp89\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050794 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.050911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.050874 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.051041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.051018 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.051117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.051076 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.051280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.051241 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.052820 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.052798 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.053102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.053072 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.061917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.061885 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.062026 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.061978 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxp89\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-kube-api-access-zxp89\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.173955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.173677 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:08.521745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.521714 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn"] Oct 31 12:41:08.523936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.523894 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:08.524042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.523988 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:08.524042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.524032 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:08.633476 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.630504 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" event={"ID":"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6","Type":"ContainerStarted","Data":"50c0e43e289e53b3f16b68e420f6dd3b95b5fc28127dd74103c393454b4be2ee"} Oct 31 12:41:08.633476 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.632177 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" event={"ID":"ff6b188d-78b4-495c-937a-9fc6e8150593","Type":"ContainerStarted","Data":"1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5"} Oct 31 12:41:08.633858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.633834 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" event={"ID":"aafa41b6-0956-45e8-8b12-481bbbcaa965","Type":"ContainerStarted","Data":"1bbacec97c73367ecde0a2a7182859f46480d1c5bdd79b37113318e2309f66b6"} Oct 31 12:41:08.633895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.633867 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" event={"ID":"aafa41b6-0956-45e8-8b12-481bbbcaa965","Type":"ContainerStarted","Data":"e776c9c553801e99c06e56692d2e84defd550d039d34b5ac6a4914bfc785adc0"} Oct 31 12:41:08.657973 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.657926 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podStartSLOduration=2.657909041 podStartE2EDuration="2.657909041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:08.655442016 +0000 UTC m=+779.470456407" watchObservedRunningTime="2026-10-31 12:41:08.657909041 +0000 UTC m=+779.472923431" Oct 31 12:41:08.720456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.720389 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podStartSLOduration=1.720368379 podStartE2EDuration="1.720368379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:08.680189299 +0000 UTC m=+779.495203695" watchObservedRunningTime="2026-10-31 12:41:08.720368379 +0000 UTC m=+779.535382770" Oct 31 12:41:08.947304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:08.944125 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:09.323543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.314927 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:41:09.323543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.314991 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:41:09.347863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.347831 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:09.613544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.612528 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:09.625492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.614296 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:41:09.625492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.614347 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:41:09.650003 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.649679 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" event={"ID":"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6","Type":"ContainerStarted","Data":"67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff"} Oct 31 12:41:09.683485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.683409 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podStartSLOduration=2.6833901669999998 podStartE2EDuration="2.683390167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:09.680964606 +0000 UTC m=+780.495978997" watchObservedRunningTime="2026-10-31 12:41:09.683390167 +0000 UTC m=+780.498404558" Oct 31 12:41:09.741488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.738844 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:41:09.760830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.760805 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:41:09.907406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:09.907318 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:10.184564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.177976 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:10.184564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.178119 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:41:10.184564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.178164 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:41:10.307922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.307886 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:41:10.308175 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.308148 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:41:10.602485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.602100 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:41:10.602485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.602160 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:41:10.755435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:10.755403 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:41:11.189492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:11.184601 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:41:11.189492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:11.184674 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:41:11.309496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:11.305793 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:41:11.309496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:11.307945 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:41:11.577543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:11.559761 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:11.742485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:11.741862 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:41:11.742485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:11.741919 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:41:12.178484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:12.177878 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:41:12.178484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:12.177939 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:41:12.305485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:12.303805 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:41:12.305485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:12.303866 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:41:12.489541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:12.489327 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:12.611485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:12.609843 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:41:12.611485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:12.609899 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:41:13.181131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:13.174915 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:41:13.181131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:13.175005 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:41:13.341853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:13.341819 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:41:13.342159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:13.342129 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:41:13.521569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:13.519381 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:13.624485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:13.622036 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:41:13.624485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:13.622113 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:41:14.178492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:14.177791 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:41:14.178492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:14.177850 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:41:14.303048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:14.302964 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:41:14.303583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:14.303032 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:41:14.468632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:14.468538 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:14.600870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:14.600835 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:41:14.601035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:14.600895 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:41:15.150997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.150961 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:15.175143 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.175107 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:41:15.175323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.175158 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:41:15.312495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.312436 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:15.313679 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.313651 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:15.314543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.314520 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 12:41:15.565984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.565869 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:15.605447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.605419 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:15.605810 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.605786 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:15.606547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:15.606522 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:41:16.296295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:16.296263 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:16.296600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:16.296578 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:16.297338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:16.297319 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 12:41:16.499123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:16.499087 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:17.315305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:17.315242 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:18.033091 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:18.033046 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:18.382402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:18.382372 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:41:18.976216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:18.976188 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:20.264486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:20.260941 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:20.987544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:20.986534 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:41:21.013054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.011958 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.156486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.150505 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.187596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.187546 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.212644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.212601 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.216567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.216530 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.230803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.230771 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.579570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.579528 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:41:21.581855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.580090 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.683482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.682541 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.894647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.894535 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.911212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.895637 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx"] Oct 31 12:41:21.948487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.945557 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:21.953678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.953648 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx"] Oct 31 12:41:21.953985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:21.953925 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.001675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.001643 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:22.002419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.002398 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:41:22.215743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.110567 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:22.215743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.110929 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:41:22.215743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.111245 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:41:22.215743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.152907 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:22.215743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.153214 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:41:22.215743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.214113 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:22.215743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.214422 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:22.267486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.221617 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:41:22.575508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.440016 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:41:22.575508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.270028 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:41:22.575508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.464805 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:41:22.575508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.560969 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:41:22.580358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.580249 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:41:22.611427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.611403 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:41:22.624707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.624676 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.624920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.624900 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.636426 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.625246 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.641929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.641899 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.642244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.642226 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtv2\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-kube-api-access-9dtv2\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.642366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.642352 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.642515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.642502 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.642627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.642612 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.642735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.642722 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.660736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.660719 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:41:22.704154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.703856 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:41:22.723280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.723237 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:41:22.762768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.762717 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.762915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.762845 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtv2\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-kube-api-access-9dtv2\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.762915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.762877 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.763017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.762923 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.763017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.762951 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.763017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.762976 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.763123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.763073 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.763156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.763145 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.763392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.763376 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.764328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.763874 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.764328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.763964 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.764328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.764142 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.764328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.764204 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.764712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.764690 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.769256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.769230 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.769555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.769529 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.806757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.806735 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtv2\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-kube-api-access-9dtv2\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.807020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.806983 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:22.880679 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.880594 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:22.911297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:22.911271 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:23.213852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.213827 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx"] Oct 31 12:41:23.215319 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:23.215297 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad04ba7_e99f_4779_95fd_e0ebd5e889ce.slice/crio-cf32ace922ea517dc4a7f602825c6a261779882ab6f899f161feeee0fc9741dd WatchSource:0}: Error finding container cf32ace922ea517dc4a7f602825c6a261779882ab6f899f161feeee0fc9741dd: Status 404 returned error can't find the container with id cf32ace922ea517dc4a7f602825c6a261779882ab6f899f161feeee0fc9741dd Oct 31 12:41:23.217938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.217904 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:23.218043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.217969 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:23.218043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.218000 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:23.437649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.437611 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" event={"ID":"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce","Type":"ContainerStarted","Data":"cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4"} Oct 31 12:41:23.437827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.437676 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" event={"ID":"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce","Type":"ContainerStarted","Data":"cf32ace922ea517dc4a7f602825c6a261779882ab6f899f161feeee0fc9741dd"} Oct 31 12:41:23.464175 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.464093 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podStartSLOduration=2.464080371 podStartE2EDuration="2.464080371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:23.459325939 +0000 UTC m=+794.274340327" watchObservedRunningTime="2026-10-31 12:41:23.464080371 +0000 UTC m=+794.279094760" Oct 31 12:41:23.777166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.777088 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:23.913049 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:23.913023 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:24.850247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:24.850213 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:24.916412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:24.916136 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 12:41:24.916412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:24.916204 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 12:41:25.855615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:25.855579 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:25.912682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:25.912649 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 12:41:25.912883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:25.912708 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 12:41:26.912161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:26.912128 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 12:41:26.912602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:26.912177 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 12:41:26.977485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:26.975112 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:27.190079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:27.189968 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:27.910488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:27.903872 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:27.911974 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:27.911944 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 12:41:27.912137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:27.912012 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 12:41:28.854853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:28.854795 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:28.912281 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:28.912239 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 12:41:28.912437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:28.912313 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 12:41:29.372870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.372818 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh"] Oct 31 12:41:29.377748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.377719 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.382142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.382113 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"distinct-2-w4-c67de2-kserve-self-signed-certs\"" Oct 31 12:41:29.397099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.397077 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh"] Oct 31 12:41:29.449138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.449115 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:29.495517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.495492 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-home\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.495696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.495666 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwj9\" (UniqueName: \"kubernetes.io/projected/3720682a-e4c6-42ff-8a5b-0daf19403f66-kube-api-access-mcwj9\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.495830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.495721 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-tmp-dir\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.495968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.495909 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-model-cache\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.496017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.495999 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-dshm\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.496063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.496044 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3720682a-e4c6-42ff-8a5b-0daf19403f66-tls-certs\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.598698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.597390 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwj9\" (UniqueName: \"kubernetes.io/projected/3720682a-e4c6-42ff-8a5b-0daf19403f66-kube-api-access-mcwj9\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.598698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.597436 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-tmp-dir\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.598698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.597643 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-model-cache\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.598698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.597737 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-dshm\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.598698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.597783 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3720682a-e4c6-42ff-8a5b-0daf19403f66-tls-certs\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.598698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.597825 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-home\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.598698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.598440 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-home\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.599281 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.599051 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-tmp-dir\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.599341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.599303 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-model-cache\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.616263 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.616235 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3720682a-e4c6-42ff-8a5b-0daf19403f66-tls-certs\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.620724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.620697 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-dshm\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.628088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.623133 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwj9\" (UniqueName: \"kubernetes.io/projected/3720682a-e4c6-42ff-8a5b-0daf19403f66-kube-api-access-mcwj9\") pod \"distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.676918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.676884 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:29.695893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.695860 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:29.912089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.912014 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 12:41:29.912089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:29.912059 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 12:41:30.010704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.010666 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh"] Oct 31 12:41:30.011966 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:30.011937 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3720682a_e4c6_42ff_8a5b_0daf19403f66.slice/crio-8a969ba499ebf1108abbe17139a1e3a0d6e78a4c89f2ad7083c6b0d33258469e WatchSource:0}: Error finding container 8a969ba499ebf1108abbe17139a1e3a0d6e78a4c89f2ad7083c6b0d33258469e: Status 404 returned error can't find the container with id 8a969ba499ebf1108abbe17139a1e3a0d6e78a4c89f2ad7083c6b0d33258469e Oct 31 12:41:30.343144 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:41:30.343103 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:41:30.529635 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.529593 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" event={"ID":"3720682a-e4c6-42ff-8a5b-0daf19403f66","Type":"ContainerStarted","Data":"2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449"} Oct 31 12:41:30.529826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.529651 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" event={"ID":"3720682a-e4c6-42ff-8a5b-0daf19403f66","Type":"ContainerStarted","Data":"8a969ba499ebf1108abbe17139a1e3a0d6e78a4c89f2ad7083c6b0d33258469e"} Oct 31 12:41:30.535451 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.535411 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:30.553571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.553524 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podStartSLOduration=1.5535112679999998 podStartE2EDuration="1.553511268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:30.551133027 +0000 UTC m=+801.366147420" watchObservedRunningTime="2026-10-31 12:41:30.553511268 +0000 UTC m=+801.368525659" Oct 31 12:41:30.699131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.699025 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:30.916303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.916275 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:30.916782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.916675 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:30.917266 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:30.917243 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:41:31.423820 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:31.423780 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:32.254346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:32.254306 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:33.314040 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:33.313893 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:34.049519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:34.049487 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:34.859296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:34.859231 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:35.539964 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:35.539930 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:36.217861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:36.217835 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:37.106032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.105993 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:37.516088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.516014 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb"] Oct 31 12:41:37.520749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.520726 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.544561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.544535 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb"] Oct 31 12:41:37.598618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.598584 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2"] Oct 31 12:41:37.604400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.604375 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.608077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.608051 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.608232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.608213 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.608305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.608255 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.608362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.608329 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76ws\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-kube-api-access-q76ws\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.608670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.608644 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.608812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.608751 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.609176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.609071 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.609285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.609187 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.609285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.609226 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.631525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.631501 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2"] Oct 31 12:41:37.680874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.680842 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4"] Oct 31 12:41:37.684816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.684792 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.711178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711151 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.711178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711188 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.711405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711301 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.711405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711356 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.711405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711390 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.711618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711452 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.711618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711532 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.711618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711603 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.711768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711632 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.711768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711673 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.711768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711722 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.711920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711786 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.711920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711888 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.712017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711956 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5szp\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-kube-api-access-p5szp\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.712017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.711997 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.712122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.712025 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.712122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.712055 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76ws\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-kube-api-access-q76ws\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.712122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.712094 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.712934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.712906 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.713778 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.713751 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.713880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.713821 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.714206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.714181 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.714301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.714255 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.715758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.715741 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.716309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.716284 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.751515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.740875 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4"] Oct 31 12:41:37.751515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.742944 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.751515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.749926 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76ws\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-kube-api-access-q76ws\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.813390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813321 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.813390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813363 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.813624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813399 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.813624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813437 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20154455-637d-4431-a426-54d6ad76f818-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.813624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813454 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.813624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813492 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.813823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813642 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.813823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813671 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5szp\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-kube-api-access-p5szp\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.813823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813700 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20154455-637d-4431-a426-54d6ad76f818-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.813823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813735 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.813823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813745 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813846 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813887 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813932 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.814086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.813974 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814068 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814114 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814144 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.814332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814186 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.814332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814253 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814278 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfjf\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-kube-api-access-xzfjf\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.814521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814404 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.814700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.814682 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.816177 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.816158 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.816309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.816293 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.825396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.825375 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.825685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.825668 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5szp\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-kube-api-access-p5szp\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.857371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.857350 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:37.916352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916318 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916382 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfjf\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-kube-api-access-xzfjf\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916485 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916602 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20154455-637d-4431-a426-54d6ad76f818-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916631 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916665 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916782 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20154455-637d-4431-a426-54d6ad76f818-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916866 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.916985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.916928 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.917433 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.917405 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.917723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.917699 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.918712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.918686 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.919130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.919106 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20154455-637d-4431-a426-54d6ad76f818-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.919225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.919207 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.920445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.920341 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.920768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.920748 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20154455-637d-4431-a426-54d6ad76f818-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.934508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.924812 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:37.934508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.932790 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:37.947187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:37.947159 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfjf\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-kube-api-access-xzfjf\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:38.006671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.004401 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:38.151861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.151826 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb"] Oct 31 12:41:38.153390 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:38.153364 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faa2b8a_979b_4b84_a35a_32d9b7413e75.slice/crio-217d5298d608dd3e01ceb8c242d317da113efc427e39b1ea978f076b4d93dc73 WatchSource:0}: Error finding container 217d5298d608dd3e01ceb8c242d317da113efc427e39b1ea978f076b4d93dc73: Status 404 returned error can't find the container with id 217d5298d608dd3e01ceb8c242d317da113efc427e39b1ea978f076b4d93dc73 Oct 31 12:41:38.156020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.155967 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.156103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.156035 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.156103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.156078 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.237152 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.237131 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2"] Oct 31 12:41:38.239009 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:38.238974 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf392b872_3c09_4dfc_9fb5_c162bad20d8e.slice/crio-36efe3b7ee9b7dc22b2d42cf08e6d9bcbe02591702694383116c181589f1f685 WatchSource:0}: Error finding container 36efe3b7ee9b7dc22b2d42cf08e6d9bcbe02591702694383116c181589f1f685: Status 404 returned error can't find the container with id 36efe3b7ee9b7dc22b2d42cf08e6d9bcbe02591702694383116c181589f1f685 Oct 31 12:41:38.242087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.242030 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.242216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.242168 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.242290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.242214 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.328910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.328810 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:38.331893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.331864 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4"] Oct 31 12:41:38.335244 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:41:38.335205 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20154455_637d_4431_a426_54d6ad76f818.slice/crio-5376df27f2568c9daa2cab808d8af53fcb1e202e3f89924738b94656d5db037f WatchSource:0}: Error finding container 5376df27f2568c9daa2cab808d8af53fcb1e202e3f89924738b94656d5db037f: Status 404 returned error can't find the container with id 5376df27f2568c9daa2cab808d8af53fcb1e202e3f89924738b94656d5db037f Oct 31 12:41:38.341237 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.340722 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.341237 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.340810 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.341237 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.340867 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:38.623246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.623164 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" event={"ID":"2faa2b8a-979b-4b84-a35a-32d9b7413e75","Type":"ContainerStarted","Data":"f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60"} Oct 31 12:41:38.623841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.623811 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" event={"ID":"2faa2b8a-979b-4b84-a35a-32d9b7413e75","Type":"ContainerStarted","Data":"217d5298d608dd3e01ceb8c242d317da113efc427e39b1ea978f076b4d93dc73"} Oct 31 12:41:38.625295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.625271 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" event={"ID":"20154455-637d-4431-a426-54d6ad76f818","Type":"ContainerStarted","Data":"3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd"} Oct 31 12:41:38.625407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.625319 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" event={"ID":"20154455-637d-4431-a426-54d6ad76f818","Type":"ContainerStarted","Data":"5376df27f2568c9daa2cab808d8af53fcb1e202e3f89924738b94656d5db037f"} Oct 31 12:41:38.628091 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.628069 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" event={"ID":"f392b872-3c09-4dfc-9fb5-c162bad20d8e","Type":"ContainerStarted","Data":"941f703504ebcb532e4e40b0711e5fb3020834439274682bd4b065775c9dc031"} Oct 31 12:41:38.628186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.628112 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" event={"ID":"f392b872-3c09-4dfc-9fb5-c162bad20d8e","Type":"ContainerStarted","Data":"36efe3b7ee9b7dc22b2d42cf08e6d9bcbe02591702694383116c181589f1f685"} Oct 31 12:41:38.678245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.678194 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podStartSLOduration=1.6781818020000001 podStartE2EDuration="1.678181802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:38.676623435 +0000 UTC m=+809.491637826" watchObservedRunningTime="2026-10-31 12:41:38.678181802 +0000 UTC m=+809.493196192" Oct 31 12:41:38.678851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.678808 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podStartSLOduration=1.678797582 podStartE2EDuration="1.678797582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:38.651049397 +0000 UTC m=+809.466063787" watchObservedRunningTime="2026-10-31 12:41:38.678797582 +0000 UTC m=+809.493811984" Oct 31 12:41:38.858080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.858033 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:38.926581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:38.925107 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:39.006876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.006838 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:39.024678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.024644 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:41:39.024910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.024883 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:41:39.388486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.385647 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:39.703545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.700887 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:39.760957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.760932 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:39.805758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.805698 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podStartSLOduration=2.805678739 podStartE2EDuration="2.805678739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:38.704929744 +0000 UTC m=+809.519944135" watchObservedRunningTime="2026-10-31 12:41:39.805678739 +0000 UTC m=+810.620693130" Oct 31 12:41:39.858932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.858899 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:41:39.859085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.858949 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:41:39.945485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.943573 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:41:39.945485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:39.943656 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:41:40.018542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.018188 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:41:40.018542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.018254 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:41:40.441577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.441045 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:40.840007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.839979 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:40.858567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.858542 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:41:40.858704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.858583 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:41:40.951521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.951446 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:41:40.951765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:40.951735 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:41:41.034487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:41.033642 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:41:41.034487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:41.033718 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:41:41.646488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:41.641063 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:41.876499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:41.859276 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:41:41.937493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:41.933532 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:41:41.974485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:41.935318 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:41:41.974765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:41.974731 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:41:42.060359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.060326 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:41:42.072291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.060627 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:41:42.584561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.582283 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:42.675023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.674627 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:42.760484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.751553 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:41:42.783551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.782153 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:42.783551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.782631 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:41:42.936541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.935355 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:41:42.936541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.935411 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:41:42.948283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.948227 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:41:42.948542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:42.948516 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:41:43.031052 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.031017 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:41:43.032222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.032192 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:41:43.260647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.260548 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:43.741486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.735241 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:43.863490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.862496 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:41:43.863490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.862555 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:41:43.955772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.955738 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:41:43.957790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:43.957739 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:41:44.021626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:44.021538 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:41:44.021907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:44.021878 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:41:44.858811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:44.858774 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:41:44.859121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:44.859073 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:41:44.925561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:44.925506 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:41:44.925730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:44.925569 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:41:44.931171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:44.931139 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:45.006094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:45.006028 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:41:45.006749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:45.006709 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:41:45.863669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:45.863585 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:45.864929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:45.864906 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:45.865647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:45.865623 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:41:45.959486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:45.958922 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:46.024656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:46.024629 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:46.026318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:46.026297 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:46.028309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:46.028290 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:41:46.240161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:46.240078 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:46.241589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:46.241552 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:46.242483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:46.242449 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:41:46.927351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:46.927321 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:48.022366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:48.022333 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:48.618656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:48.618621 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:48.835480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:48.835433 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:49.796849 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:49.796774 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:50.614506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:50.614425 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:51.421964 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:51.421932 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:52.373493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.373234 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94"] Oct 31 12:41:52.417550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.414328 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94"] Oct 31 12:41:52.417550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.414516 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.587874 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.587965 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmmp\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-kube-api-access-czmmp\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.588051 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.588089 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.588142 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.588174 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.588209 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.588324 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.592486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.588355 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691198 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691248 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691407 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691520 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmmp\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-kube-api-access-czmmp\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691654 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691696 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691752 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691786 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.691822 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.692305 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.701578 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.701903 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.702133 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.717532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.702664 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.800489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.797009 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.830982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.815530 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmmp\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-kube-api-access-czmmp\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.843981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.843948 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.858494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.844640 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:52.903746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:52.903455 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:53.014737 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.014686 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:53.606486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.600795 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:53.637490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.636615 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:53.656486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.645686 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:53.675485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.670683 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:41:53.687573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.687532 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:41:53.779595 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.779552 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:53.905485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:53.903558 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:54.123742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:54.120098 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:41:54.606578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:54.606535 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 31 12:41:54.632487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:54.607033 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:55.018569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:54.952816 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:41:55.018569 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.012177 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:41:55.041206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.040902 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:41:55.047920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.042515 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:41:55.047920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.042917 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:41:55.047920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.043250 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:41:55.047920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.043955 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:41:55.047920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.044545 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:41:55.092751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.092639 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.128732 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.129324 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.130149 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.130240 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.130396 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.130477 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.130542 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.130676 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.130739 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:41:55.190525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.157990 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:41:55.334556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.271094 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:41:55.334556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.271157 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94"] Oct 31 12:41:55.334556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.318602 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:55.716833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.716724 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:41:55.814518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.798884 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:55.814518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.799072 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:55.905085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:55.905035 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:41:56.117343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.117285 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:41:56.120485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.120440 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:56.121967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.121949 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:41:56.150912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.150870 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:56.155559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.155526 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:56.160073 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.160045 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:56.167571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.167544 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:41:56.191667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.191627 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:41:56.295565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.270135 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:56.351484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.326184 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:41:56.451805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.451760 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:41:56.452253 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.452226 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:56.582504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.571428 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:41:56.582504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.571520 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:41:56.582504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.571572 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:41:56.788486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.785118 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" podUID="bfa9cd02-a108-4237-bc59-eb14f393cd2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:56.994488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:56.976656 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:41:57.135357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.135312 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:57.413279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.151559 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:41:57.487784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.151657 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:57.488838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.190858 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:57.489840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.190891 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:57.490442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.190811 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:57.494080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.192594 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:41:57.494654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.306765 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:41:57.494784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.397760 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:57.495350 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.456839 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:41:57.495685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.480808 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:57.711970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:57.711940 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" event={"ID":"6f85dc17-8edb-4c31-92a7-99afba1dc8c6","Type":"ContainerStarted","Data":"a78f3e4f42613310cab44ab70b617a7d6bc052138c7698bb4d67452437522bef"} Oct 31 12:41:58.000643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.000549 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:58.411640 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.411583 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:41:58.504500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.502548 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:58.504787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.504746 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.505552 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.508762 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.508995 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.509207 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.509644 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.509957 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.511220 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:41:58.521490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.512063 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:41:58.583481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.581324 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:41:58.623482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.613906 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:41:58.659228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.642867 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:41:58.659228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.645349 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:58.659228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.648732 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:41:58.659228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.654439 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" event={"ID":"6f85dc17-8edb-4c31-92a7-99afba1dc8c6","Type":"ContainerStarted","Data":"a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960"} Oct 31 12:41:58.682742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.660538 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:41:58.682742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.671677 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:41:58.684019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.683997 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:41:58.726487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.721879 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:41:58.738697 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.738670 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:41:58.738822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.738728 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:41:58.738822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.738767 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:41:58.738953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.738838 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:41:58.795550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.795497 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podStartSLOduration=6.79547796 podStartE2EDuration="6.79547796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:58.792726041 +0000 UTC m=+829.607740448" watchObservedRunningTime="2026-10-31 12:41:58.79547796 +0000 UTC m=+829.610492352" Oct 31 12:41:58.904031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:58.904000 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:41:59.563489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:59.560959 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:59.905541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:59.905500 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 12:41:59.905739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:41:59.905559 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 12:42:00.342807 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:42:00.342768 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:42:00.546484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:00.543826 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:00.904769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:00.904735 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 12:42:00.905198 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:00.904799 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 12:42:01.343817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:01.343791 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:01.904854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:01.904822 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 12:42:01.905234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:01.904879 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 12:42:02.142785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:02.142753 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:02.905005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:02.904962 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 12:42:02.905477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:02.905022 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 12:42:02.936144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:02.936116 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:03.710813 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:03.710781 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:03.904409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:03.904374 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 12:42:03.904615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:03.904444 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 12:42:04.638278 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:04.638244 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:04.905185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:04.905109 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 12:42:04.905323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:04.905180 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 12:42:05.548748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:05.542368 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:05.908790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:05.908760 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:42:05.909153 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:05.909054 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:42:05.909885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:05.909868 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:42:06.470223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:06.470179 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:06.712475 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:06.712425 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:07.230502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:07.230446 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:07.991714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:07.991678 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47"] Oct 31 12:42:07.996198 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:07.996178 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.015776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.015752 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47"] Oct 31 12:42:08.069052 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.069023 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:08.121368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121331 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121370 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8qd\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-kube-api-access-2c8qd\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121560 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121607 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121744 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121723 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441559e3-5ae0-40bb-b7cc-604837dea088-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121755 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121783 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121932 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.121981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.121957 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441559e3-5ae0-40bb-b7cc-604837dea088-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223004 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223064 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8qd\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-kube-api-access-2c8qd\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223171 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223201 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223301 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441559e3-5ae0-40bb-b7cc-604837dea088-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223333 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223380 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223529 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223553 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223555 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441559e3-5ae0-40bb-b7cc-604837dea088-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223635 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223777 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.223883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.223826 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.224138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.224117 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441559e3-5ae0-40bb-b7cc-604837dea088-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.225871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.225845 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.225976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.225894 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441559e3-5ae0-40bb-b7cc-604837dea088-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.235161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.235138 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.235451 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.235249 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8qd\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-kube-api-access-2c8qd\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.314233 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.314172 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:08.603663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.603637 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47"] Oct 31 12:42:08.606392 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:42:08.606369 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441559e3_5ae0_40bb_b7cc_604837dea088.slice/crio-0c4f1050cbd7091520eceff49c88f3cddca34d0dbb49693208fd0584046c6d18 WatchSource:0}: Error finding container 0c4f1050cbd7091520eceff49c88f3cddca34d0dbb49693208fd0584046c6d18: Status 404 returned error can't find the container with id 0c4f1050cbd7091520eceff49c88f3cddca34d0dbb49693208fd0584046c6d18 Oct 31 12:42:08.608362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.608330 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:08.608442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.608401 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:08.608442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.608431 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:08.755990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.755956 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" event={"ID":"441559e3-5ae0-40bb-b7cc-604837dea088","Type":"ContainerStarted","Data":"bcad8df32898d0d5792ee11c8b957d77f2da3a6975508768bc92f106b5f4402b"} Oct 31 12:42:08.756174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.756015 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" event={"ID":"441559e3-5ae0-40bb-b7cc-604837dea088","Type":"ContainerStarted","Data":"0c4f1050cbd7091520eceff49c88f3cddca34d0dbb49693208fd0584046c6d18"} Oct 31 12:42:08.782168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.782122 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podStartSLOduration=1.78210794 podStartE2EDuration="1.78210794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:08.778262429 +0000 UTC m=+839.593276824" watchObservedRunningTime="2026-10-31 12:42:08.78210794 +0000 UTC m=+839.597122335" Oct 31 12:42:08.955339 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:08.955227 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:09.314448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:09.314377 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:09.315956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:09.315919 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:42:09.316105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:09.315987 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 12:42:09.987486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:09.983018 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:10.314732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:10.314654 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:42:10.315157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:10.314721 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 12:42:10.916534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:10.916502 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:11.315235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:11.315157 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:42:11.315235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:11.315215 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 12:42:11.647633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:11.647607 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:12.314448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:12.314417 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:42:12.314632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:12.314493 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 12:42:12.485499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:12.482671 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:13.273835 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:13.273800 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:13.316064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:13.315420 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:42:13.316064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:13.315522 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 12:42:14.185428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:14.185393 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:14.315529 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:14.315498 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:42:14.315675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:14.315546 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 12:42:14.923014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:14.922973 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:15.314947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:15.314877 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:42:15.314947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:15.314926 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 12:42:15.730040 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:15.730000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:16.319179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:16.319148 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:16.319698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:16.319570 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:16.320133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:16.320117 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:42:16.633487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:16.628640 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:17.546078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:17.544485 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:18.269409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:18.269365 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:19.235774 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:19.235737 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:19.854335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:19.854306 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:42:20.043276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:20.043229 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:20.886996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:20.886959 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:21.296296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:21.296202 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:22.072487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.054329 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml"] Oct 31 12:42:22.077558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.076718 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.136524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.095444 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml"] Oct 31 12:42:22.171499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.156573 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:22.195923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.195564 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.262679 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.262763 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.262818 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcwt\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-kube-api-access-qkcwt\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.262869 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.262929 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.263027 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.263054 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.263097 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.270500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.263200 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.494570 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.451617 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcwt\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-kube-api-access-qkcwt\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.496636 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.496705 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.496818 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.496846 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.496893 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.497011 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.497189 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.497261 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.497853 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.498134 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.498381 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.547992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.518965 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.571493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.554915 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:22.571493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.555217 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:22.571493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.561535 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:42:22.571493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.561939 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:22.571493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.562216 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:42:22.624487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.606848 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:22.637497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.631058 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:22.637497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.631882 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.804230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.685640 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb"] Oct 31 12:42:22.804230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.690119 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:22.804230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.762557 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:22.804230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.791865 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:22.888489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.862165 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:22.968489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.960017 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.981915 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdxg\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-kube-api-access-2wdxg\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.981993 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.982091 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.982190 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.982219 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.982314 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.982364 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.982427 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:22.982494 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.076009 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb"] Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.076138 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.088786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.079228 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.088821 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.088886 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.088950 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.089003 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.089140 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdxg\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-kube-api-access-2wdxg\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.089192 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.089264 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.089374 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.089838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.089401 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.090363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.089895 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.101490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.098296 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.101490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.099036 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.101490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.099567 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.128343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.128312 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.135278 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.134849 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.160654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.157891 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.160654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.159344 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:42:23.187294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.187242 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.205505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.199016 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:23.206391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.206368 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:42:23.221629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.207860 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 12:42:23.510352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.510252 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.533151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.533108 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.583870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.583823 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.625488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.584102 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:42:23.625488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.584139 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:23.625488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.584159 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:23.625488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.584186 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.625488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.608550 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.641491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.634273 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcwt\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-kube-api-access-qkcwt\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:23.641491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.634363 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:23.952489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.925191 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.952489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.925647 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:23.952489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.933690 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdxg\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-kube-api-access-2wdxg\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:23.952489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.948286 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:24.117579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.954419 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.162560 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.073732 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.164030 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.164072 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.164109 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.164147 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.164179 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.164212 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.164245 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:42:24.165488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.094013 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:24.203547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:23.963358 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:42:24.205188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.205149 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:24.219156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.219130 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:42:24.219433 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.219416 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.607160 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.699545 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.699912 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.700186 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.700425 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.700704 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.700926 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:24.746182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.701166 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:24.772604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:24.772558 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:42:25.091654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.091598 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 31 12:42:25.412437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.412392 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:42:25.485307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.449683 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:42:25.485307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.449732 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:42:25.485307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.449754 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:25.606487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.603046 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.813603 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.813897 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.814122 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.814286 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.814455 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.814642 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.814807 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:42:25.840548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:25.794406 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:26.487259 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.487211 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:42:26.652824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.652785 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.793249 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.799969 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.814802 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.815117 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.815381 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.815643 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.815880 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.822789 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.829525 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:26.951545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:26.832995 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:27.490492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.489579 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:42:27.663825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.663763 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:27.803542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.802560 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:27.857493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.857429 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:42:27.861609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.861564 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:27.875493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.869033 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:27.875493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.869363 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:27.875493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.869645 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:42:27.875493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.869899 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:42:27.875493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.870776 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:42:27.894527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.880917 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:27.895324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:27.895291 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:28.265011 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.264865 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:28.643884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.591746 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:42:28.744409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.743735 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:28.918549 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.918414 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:28.930415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.929846 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:28.987477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.965923 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:42:28.987477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.976730 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml"] Oct 31 12:42:28.987477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.981384 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:28.987477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.983222 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:42:28.987477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.983786 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:28.987477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.984276 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:42:28.987477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.984334 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:28.988963 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.988790 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:28.989077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:28.989052 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:29.014953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.014930 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:42:29.015191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.015178 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:42:29.023866 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.023824 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:29.027479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.024104 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:29.027744 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.027718 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:29.063817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.063796 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:42:29.072398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.072373 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:42:29.074690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.074669 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:42:29.083435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.082875 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:42:29.083435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.083367 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:42:29.087806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.087785 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:42:29.088131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.088114 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:42:29.089970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.089949 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:42:29.096354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.096315 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:42:29.113825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.113792 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" event={"ID":"25e7214c-ed96-4ac6-a2b4-e7e954ce2673","Type":"ContainerStarted","Data":"c6533498b5e77583653e85c6a40797b9bf8e7f59ddf3069ee9439675998f7a38"} Oct 31 12:42:29.121969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.121751 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb"] Oct 31 12:42:29.123615 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:42:29.123575 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fe32e7_0e9b_4566_bb27_1d64dd1b093c.slice/crio-0933f6207ee2743dcc4d517fc128ab4f409018ffb92a1651382f6f2d51930a96 WatchSource:0}: Error finding container 0933f6207ee2743dcc4d517fc128ab4f409018ffb92a1651382f6f2d51930a96: Status 404 returned error can't find the container with id 0933f6207ee2743dcc4d517fc128ab4f409018ffb92a1651382f6f2d51930a96 Oct 31 12:42:29.126232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.126202 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:29.126325 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.126286 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:29.126396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.126351 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:29.860996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:29.860963 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:30.133542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.132904 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" event={"ID":"25e7214c-ed96-4ac6-a2b4-e7e954ce2673","Type":"ContainerStarted","Data":"de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4"} Oct 31 12:42:30.134776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.134751 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" event={"ID":"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c","Type":"ContainerStarted","Data":"c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21"} Oct 31 12:42:30.134915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.134810 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" event={"ID":"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c","Type":"ContainerStarted","Data":"0933f6207ee2743dcc4d517fc128ab4f409018ffb92a1651382f6f2d51930a96"} Oct 31 12:42:30.223486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.212348 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podStartSLOduration=9.212328035 podStartE2EDuration="9.212328035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:30.170981237 +0000 UTC m=+860.985995630" watchObservedRunningTime="2026-10-31 12:42:30.212328035 +0000 UTC m=+861.027342425" Oct 31 12:42:30.223486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.218190 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podStartSLOduration=8.218170367 podStartE2EDuration="8.218170367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:30.212104716 +0000 UTC m=+861.027119106" watchObservedRunningTime="2026-10-31 12:42:30.218170367 +0000 UTC m=+861.033184759" Oct 31 12:42:30.346813 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:42:30.346775 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:42:30.954603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.954569 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:30.955455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.955436 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 12:42:30.955613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.955589 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 12:42:30.979061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.978887 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:42:30.979061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.978942 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 12:42:30.979061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:30.979005 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:31.317546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:31.305431 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:31.955352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:31.955319 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 12:42:31.955740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:31.955366 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 12:42:32.019771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:32.017734 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:42:32.019771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:32.017814 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 12:42:32.429532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:32.426964 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:32.955533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:32.955500 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 12:42:32.955902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:32.955551 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 12:42:32.971795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:32.971762 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:42:32.972037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:32.972009 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 12:42:33.414415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:33.414367 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:33.954649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:33.954618 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 12:42:33.954814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:33.954671 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 12:42:33.966782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:33.966749 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:42:33.967327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:33.967299 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 12:42:34.282910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:34.282823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:34.955384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:34.955351 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 12:42:34.955589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:34.955403 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 12:42:34.966804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:34.966778 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:42:34.967325 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:34.967298 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 12:42:35.087892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:35.087859 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:35.911118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:35.911060 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:35.954840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:35.954801 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 12:42:35.955141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:35.955107 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 12:42:35.964108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:35.964072 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:42:35.964215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:35.964123 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 12:42:36.756661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:36.756634 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:36.959407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:36.959373 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:36.959796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:36.959773 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:36.960439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:36.960421 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:42:36.967686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:36.967667 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:36.967941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:36.967927 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:36.968545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:36.968528 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 12:42:37.622989 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.622960 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:37.856878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.856849 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw"] Oct 31 12:42:37.861348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.861324 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.886243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.886171 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw"] Oct 31 12:42:37.958350 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.958319 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.958617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.958481 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.958691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.958616 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn546\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-kube-api-access-sn546\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.958826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.958805 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.958888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.958841 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.959057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.959032 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.959170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.959132 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97f8939-24d3-4f00-afd1-1238f8a59e98-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.959217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.959164 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:37.959258 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:37.959241 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060211 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060296 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060364 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn546\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-kube-api-access-sn546\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060551 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060581 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060657 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060734 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97f8939-24d3-4f00-afd1-1238f8a59e98-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060774 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.060862 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.060816 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.061031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.061005 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.061120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.061058 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.061266 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.061242 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.061323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.061256 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.061363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.061339 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97f8939-24d3-4f00-afd1-1238f8a59e98-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.062618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.062590 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.063060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.062870 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.082239 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.082216 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn546\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-kube-api-access-sn546\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.084588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.084564 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.175837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.175762 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:38.468120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.468090 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw"] Oct 31 12:42:38.468949 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:42:38.468914 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97f8939_24d3_4f00_afd1_1238f8a59e98.slice/crio-d5925fdb7e76c79b28a20aaf6ea21cfc687b45e26d8c934b53caf859c954b70f WatchSource:0}: Error finding container d5925fdb7e76c79b28a20aaf6ea21cfc687b45e26d8c934b53caf859c954b70f: Status 404 returned error can't find the container with id d5925fdb7e76c79b28a20aaf6ea21cfc687b45e26d8c934b53caf859c954b70f Oct 31 12:42:38.471445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.471410 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:38.471556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.471505 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:38.471619 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.471556 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:38.534579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:38.534550 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:39.249688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:39.249649 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" event={"ID":"a97f8939-24d3-4f00-afd1-1238f8a59e98","Type":"ContainerStarted","Data":"e6c9312b1a42751430aa0f0fe2311fd978d9c95ed8399c622aee57834825c42b"} Oct 31 12:42:39.250325 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:39.250300 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" event={"ID":"a97f8939-24d3-4f00-afd1-1238f8a59e98","Type":"ContainerStarted","Data":"d5925fdb7e76c79b28a20aaf6ea21cfc687b45e26d8c934b53caf859c954b70f"} Oct 31 12:42:39.281819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:39.281757 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podStartSLOduration=2.281736644 podStartE2EDuration="2.281736644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:39.279963011 +0000 UTC m=+870.094977401" watchObservedRunningTime="2026-10-31 12:42:39.281736644 +0000 UTC m=+870.096751035" Oct 31 12:42:39.440701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:39.440671 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:40.178813 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:40.178774 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:40.189755 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:40.189723 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:42:40.189919 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:40.189775 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:42:40.540024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:40.539946 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:41.188487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:41.187564 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:42:41.188487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:41.187627 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:42:41.533481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:41.533367 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:42.184486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:42.184427 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:42:42.184972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:42.184512 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:42:42.334216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:42.334186 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:42.588873 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:42.588834 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:42.728866 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:42.728835 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:43.186488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:43.183982 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:42:43.186488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:43.184054 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:42:43.288923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:43.288886 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:43.437201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:43.437118 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:44.179486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:44.177786 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:42:44.179486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:44.177849 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:42:44.318666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:44.318634 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:45.163153 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:45.163117 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:45.176256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:45.176225 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:42:45.176426 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:45.176326 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:42:46.073733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.073691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:46.181483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.181441 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:46.181863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.181842 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:46.182552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.182528 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:42:46.355922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.355835 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:46.743231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.743149 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz"] Oct 31 12:42:46.748786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.748753 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.760192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.759100 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-59fd8a28-openshift-default-dockercfg-5x6c2\"" Oct 31 12:42:46.771491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.771449 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz"] Oct 31 12:42:46.845071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.845044 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:46.884956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.884928 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-data\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.884958 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-token\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.884994 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-certs\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885152 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.885084 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54glf\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-kube-api-access-54glf\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.885167 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-credential-socket\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.885219 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-envoy\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.885322 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-socket\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.885373 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istiod-ca-cert\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.885447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.885437 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-podinfo\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.975314 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.975280 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:46.987540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987502 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-envoy\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.987768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987616 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-socket\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.987768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987662 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istiod-ca-cert\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.987768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987758 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-podinfo\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.988030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987867 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-data\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.988030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987892 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-token\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.988030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987948 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-certs\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.988030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.987979 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54glf\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-kube-api-access-54glf\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.988285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.988061 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-credential-socket\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.988610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.988582 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-credential-socket\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.989685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.989659 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-data\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.994318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.993318 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-certs\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.994456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.993585 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-socket\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:46.994579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:46.994231 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istiod-ca-cert\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:47.025634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.025565 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-envoy\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:47.038568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.026563 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54glf\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-kube-api-access-54glf\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:47.051496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.041205 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-podinfo\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:47.051496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.041670 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-token\") pod \"e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:47.068005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.067751 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:47.418496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.418454 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz"] Oct 31 12:42:47.419799 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:42:47.419763 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd869ac_e11b_44fd_b55e_90d4e7a591d6.slice/crio-f3460beddb1716a800f302f454ba2c2b14de9b5646e4d0d1f7acb15cb12300d1 WatchSource:0}: Error finding container f3460beddb1716a800f302f454ba2c2b14de9b5646e4d0d1f7acb15cb12300d1: Status 404 returned error can't find the container with id f3460beddb1716a800f302f454ba2c2b14de9b5646e4d0d1f7acb15cb12300d1 Oct 31 12:42:47.422596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.422555 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:47.422688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.422642 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:47.422688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.422679 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:42:47.975827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:47.974158 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:48.379583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:48.377744 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" event={"ID":"afd869ac-e11b-44fd-b55e-90d4e7a591d6","Type":"ContainerStarted","Data":"944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0"} Oct 31 12:42:48.379583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:48.377819 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" event={"ID":"afd869ac-e11b-44fd-b55e-90d4e7a591d6","Type":"ContainerStarted","Data":"f3460beddb1716a800f302f454ba2c2b14de9b5646e4d0d1f7acb15cb12300d1"} Oct 31 12:42:49.166582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.164850 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:49.206839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.206799 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:49.270741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.270676 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" podStartSLOduration=3.270657508 podStartE2EDuration="3.270657508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:48.42402251 +0000 UTC m=+879.239036900" watchObservedRunningTime="2026-10-31 12:42:49.270657508 +0000 UTC m=+880.085671899" Oct 31 12:42:49.526528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.526427 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:49.544257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.544226 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:42:49.569489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.562355 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:42:49.569489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.562301 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:49.643488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.639826 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:49.844486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.842930 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:42:49.883489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.879538 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:49.883489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.879931 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:49.927488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:49.914492 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.017501 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.001568 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.026509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.026167 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:50.167562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.167435 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.223289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.181091 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.223289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.181522 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.235048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.233588 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:50.308559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.306821 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:50.480542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.474116 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.579488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.572547 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.579488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.575683 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:42:50.616488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.598574 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:42:50.616488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.598655 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:42:50.900594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.851568 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:50.900594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.851899 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:42:50.900594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.852274 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:42:50.964495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.946561 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:50.964495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.947231 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:42:50.980578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.968073 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:51.009511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.971777 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.009511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:50.971866 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 31 12:42:51.056545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.040777 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.087555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.087518 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:42:51.089586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.089546 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:42:51.094617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.090547 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:42:51.300512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.217395 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:42:51.300512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.219322 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:42:51.300512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.220325 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.300512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.220418 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.300512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.254871 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.300512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.257177 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:42:51.300512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.257268 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:42:51.324921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.313338 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:51.324921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.315255 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:51.482780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.482737 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.634868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.572758 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.636382 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.587558 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.903945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.903838 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:51.967871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:51.967821 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:52.037991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.037933 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 31 12:42:52.101492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.098437 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:52.120498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.120442 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:42:52.120752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.120724 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:52.148783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.148740 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:52.171749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.150410 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:52.328487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.322149 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:42:52.449866 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.449749 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:52.550743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.550693 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:52.702641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.702542 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:42:52.708591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.703162 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:42:52.729736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.709076 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:42:52.829019 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:42:52.813443 6727 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.487s" Oct 31 12:42:52.829019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.813634 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:42:52.829019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.813675 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:42:52.875645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.786291 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:42:52.875979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.722911 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:52.917516 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.912815 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 31 12:42:52.919633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:52.919593 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:53.162794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.131833 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 31 12:42:53.162794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.131939 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:53.162794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.160371 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:42:53.162794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.160428 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:53.193532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.172843 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:53.193532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.174236 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:53.193532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.189321 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:42:53.263839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.263803 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:53.453106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.453002 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:53.493487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.492557 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 31 12:42:53.493487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.492649 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:42:53.497486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.472543 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" podUID="bfa9cd02-a108-4237-bc59-eb14f393cd2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 31 12:42:53.586042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.585924 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 12:42:53.661125 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.661094 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:53.771538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.759535 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:42:53.895031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.769843 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:53.895031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.770272 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:53.895031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.882575 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:42:53.903743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.815157 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:53.946515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:53.921836 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:54.311652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.311612 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:42:54.474860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.427915 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:42:54.474860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.436204 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:42:54.474860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.440003 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:42:54.745133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.745087 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:42:54.746126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.746094 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:42:54.775557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.767436 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 31 12:42:54.817815 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.798709 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:42:54.834313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.830984 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:54.911795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.911743 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:42:54.912340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.912313 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:42:54.940565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.940519 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:54.960233 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.960203 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:54.968303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.963909 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:54.968760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:54.968183 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:55.258879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.258849 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:42:55.286663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.286617 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v"] Oct 31 12:42:55.353601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.353560 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 31 12:42:55.398556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.398520 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:55.460783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.460737 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:55.603560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.592665 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:42:55.624368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.611098 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:42:55.624368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.611940 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:55.624368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.615775 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-59fd8a28\"" Oct 31 12:42:55.677354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.677332 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:42:55.682114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.678568 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:42:55.682114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.678611 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:42:55.682114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.678630 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v"] Oct 31 12:42:55.682114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.678835 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:42:55.682114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.681588 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:42:55.682114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.681626 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:42:55.714586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.713735 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:42:55.714995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.714970 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:42:55.717353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.717328 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:42:55.783169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.783122 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9627a91-c210-48bc-a39e-40429c01645a-plugins-config-volume\") pod \"payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:55.783356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.783323 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dds\" (UniqueName: \"kubernetes.io/projected/d9627a91-c210-48bc-a39e-40429c01645a-kube-api-access-99dds\") pod \"payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:55.887596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.885695 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9627a91-c210-48bc-a39e-40429c01645a-plugins-config-volume\") pod \"payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:55.887596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.885815 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dds\" (UniqueName: \"kubernetes.io/projected/d9627a91-c210-48bc-a39e-40429c01645a-kube-api-access-99dds\") pod \"payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:55.887596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.886852 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9627a91-c210-48bc-a39e-40429c01645a-plugins-config-volume\") pod \"payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:55.909083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.909052 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dds\" (UniqueName: \"kubernetes.io/projected/d9627a91-c210-48bc-a39e-40429c01645a-kube-api-access-99dds\") pod \"payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:55.984340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:55.984295 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:56.036203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:56.036142 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:56.283543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:56.282882 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:56.411651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:56.409980 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v"] Oct 31 12:42:56.592304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:56.592272 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" event={"ID":"d9627a91-c210-48bc-a39e-40429c01645a","Type":"ContainerStarted","Data":"05343067a8408e6c5e0c74ec5facb75882bc83b3decba772ce6cfdae31f278f2"} Oct 31 12:42:56.592485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:56.592319 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" event={"ID":"d9627a91-c210-48bc-a39e-40429c01645a","Type":"ContainerStarted","Data":"1bba515d9b490ee98c09132e3dacac2781fc71a4e6fe26b1953ecb97ea45b6aa"} Oct 31 12:42:56.592485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:56.592411 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:42:56.621819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:56.621773 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" podStartSLOduration=2.621759592 podStartE2EDuration="2.621759592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:56.618155388 +0000 UTC m=+887.433169780" watchObservedRunningTime="2026-10-31 12:42:56.621759592 +0000 UTC m=+887.436773981" Oct 31 12:42:57.145458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:57.145433 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:57.863753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:57.863723 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:58.874954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:58.874924 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:59.414647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:59.414616 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:59.557996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:59.557964 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:59.875213 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:42:59.875183 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:00.347026 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:00.346988 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:43:00.435221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.435172 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:00.616691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.616611 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:00.764783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.764748 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7"] Oct 31 12:43:00.769412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.769389 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:00.782704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.782678 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d5e8c3f2\"" Oct 31 12:43:00.782826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.782761 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d5e8c3f2-dockercfg-clwb8\"" Oct 31 12:43:00.786367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.786342 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7"] Oct 31 12:43:00.819930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.819904 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:00.835978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.835955 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn"] Oct 31 12:43:00.840302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.840284 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:00.861373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.861349 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn"] Oct 31 12:43:00.908514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.908258 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4577g\" (UniqueName: \"kubernetes.io/projected/d87dc592-2996-452d-b4de-6311df85a230-kube-api-access-4577g\") pod \"payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:00.909126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.909082 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xsj\" (UniqueName: \"kubernetes.io/projected/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-kube-api-access-j5xsj\") pod \"payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:00.909436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.909406 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:00.910354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.909571 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87dc592-2996-452d-b4de-6311df85a230-plugins-config-volume\") pod \"payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:00.993573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:00.990026 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:01.013662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.012625 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4577g\" (UniqueName: \"kubernetes.io/projected/d87dc592-2996-452d-b4de-6311df85a230-kube-api-access-4577g\") pod \"payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:01.013927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.013902 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xsj\" (UniqueName: \"kubernetes.io/projected/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-kube-api-access-j5xsj\") pod \"payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:01.014210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.014191 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:01.014364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.014319 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87dc592-2996-452d-b4de-6311df85a230-plugins-config-volume\") pod \"payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:01.017346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.015653 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87dc592-2996-452d-b4de-6311df85a230-plugins-config-volume\") pod \"payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:01.018845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.018824 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:01.101898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.088794 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4577g\" (UniqueName: \"kubernetes.io/projected/d87dc592-2996-452d-b4de-6311df85a230-kube-api-access-4577g\") pod \"payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:01.101898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.097260 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xsj\" (UniqueName: \"kubernetes.io/projected/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-kube-api-access-j5xsj\") pod \"payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:01.157118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.157070 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:01.179912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.179817 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:01.384679 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.384646 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:01.501845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.501633 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn"] Oct 31 12:43:01.630556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.630526 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" event={"ID":"d87dc592-2996-452d-b4de-6311df85a230","Type":"ContainerStarted","Data":"6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a"} Oct 31 12:43:01.630702 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.630579 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" event={"ID":"d87dc592-2996-452d-b4de-6311df85a230","Type":"ContainerStarted","Data":"a4b49eb649831214c1da0de2d8c08407a165fdc6221fccc92dfc1f98a704910f"} Oct 31 12:43:01.630702 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.630602 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:01.688656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.688613 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" podStartSLOduration=1.688600284 podStartE2EDuration="1.688600284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:01.654879561 +0000 UTC m=+892.469893952" watchObservedRunningTime="2026-10-31 12:43:01.688600284 +0000 UTC m=+892.503614673" Oct 31 12:43:01.688981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.688966 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7"] Oct 31 12:43:01.689694 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:01.689669 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5694ad_aec8_4ad5_8ce5_d93e42becd6e.slice/crio-d71eb6915e715b128ac6e2f44ff15efee2cac003cac9080036a1ddff5419f982 WatchSource:0}: Error finding container d71eb6915e715b128ac6e2f44ff15efee2cac003cac9080036a1ddff5419f982: Status 404 returned error can't find the container with id d71eb6915e715b128ac6e2f44ff15efee2cac003cac9080036a1ddff5419f982 Oct 31 12:43:01.990001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:01.989921 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:02.019700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.019673 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj"] Oct 31 12:43:02.024236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.024213 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.031696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.031667 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b849a4c4\"" Oct 31 12:43:02.031813 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.031767 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b849a4c4-dockercfg-7tksg\"" Oct 31 12:43:02.036098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.036077 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj"] Oct 31 12:43:02.081604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.081581 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw"] Oct 31 12:43:02.086151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.086129 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.109126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.109087 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw"] Oct 31 12:43:02.166487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.149925 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f1404-c7a4-441e-884f-6559609be9aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.166487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.149982 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca75aad-681e-48c3-9c57-7f2debcb4925-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.166487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.150124 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjnt\" (UniqueName: \"kubernetes.io/projected/bd0f1404-c7a4-441e-884f-6559609be9aa-kube-api-access-9mjnt\") pod \"payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.166487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.150243 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfxf\" (UniqueName: \"kubernetes.io/projected/aca75aad-681e-48c3-9c57-7f2debcb4925-kube-api-access-bdfxf\") pod \"payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.213131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.213098 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:02.254044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.251882 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f1404-c7a4-441e-884f-6559609be9aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.254044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.251946 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca75aad-681e-48c3-9c57-7f2debcb4925-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.254044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.252062 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjnt\" (UniqueName: \"kubernetes.io/projected/bd0f1404-c7a4-441e-884f-6559609be9aa-kube-api-access-9mjnt\") pod \"payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.254044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.252186 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfxf\" (UniqueName: \"kubernetes.io/projected/aca75aad-681e-48c3-9c57-7f2debcb4925-kube-api-access-bdfxf\") pod \"payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.256308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.255834 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f1404-c7a4-441e-884f-6559609be9aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.257220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.256641 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca75aad-681e-48c3-9c57-7f2debcb4925-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.286504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.280884 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjnt\" (UniqueName: \"kubernetes.io/projected/bd0f1404-c7a4-441e-884f-6559609be9aa-kube-api-access-9mjnt\") pod \"payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.297112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.297078 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfxf\" (UniqueName: \"kubernetes.io/projected/aca75aad-681e-48c3-9c57-7f2debcb4925-kube-api-access-bdfxf\") pod \"payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.343119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.342840 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:02.403736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.402818 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:02.652444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.652410 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" event={"ID":"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e","Type":"ContainerStarted","Data":"9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f"} Oct 31 12:43:02.652692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.652516 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" event={"ID":"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e","Type":"ContainerStarted","Data":"d71eb6915e715b128ac6e2f44ff15efee2cac003cac9080036a1ddff5419f982"} Oct 31 12:43:02.652692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.652679 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:02.683392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.683342 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" podStartSLOduration=2.6833297050000002 podStartE2EDuration="2.683329705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:02.68009982 +0000 UTC m=+893.495114222" watchObservedRunningTime="2026-10-31 12:43:02.683329705 +0000 UTC m=+893.498344093" Oct 31 12:43:02.699952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.699926 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj"] Oct 31 12:43:02.700497 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:02.700458 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca75aad_681e_48c3_9c57_7f2debcb4925.slice/crio-826fe5cb02925f723e857f7d0160d3a417870e8f90822bf99e7ed097e9851868 WatchSource:0}: Error finding container 826fe5cb02925f723e857f7d0160d3a417870e8f90822bf99e7ed097e9851868: Status 404 returned error can't find the container with id 826fe5cb02925f723e857f7d0160d3a417870e8f90822bf99e7ed097e9851868 Oct 31 12:43:02.732940 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.732922 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw"] Oct 31 12:43:02.733451 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:02.733428 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0f1404_c7a4_441e_884f_6559609be9aa.slice/crio-0e954f417aeb354bee27cde39e63e6696478332e3ed37649094d826c794643a1 WatchSource:0}: Error finding container 0e954f417aeb354bee27cde39e63e6696478332e3ed37649094d826c794643a1: Status 404 returned error can't find the container with id 0e954f417aeb354bee27cde39e63e6696478332e3ed37649094d826c794643a1 Oct 31 12:43:02.923988 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:02.923945 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:03.082132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.082026 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:03.376981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.376889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:03.668429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.668343 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" event={"ID":"bd0f1404-c7a4-441e-884f-6559609be9aa","Type":"ContainerStarted","Data":"5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26"} Oct 31 12:43:03.668429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.668413 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" event={"ID":"bd0f1404-c7a4-441e-884f-6559609be9aa","Type":"ContainerStarted","Data":"0e954f417aeb354bee27cde39e63e6696478332e3ed37649094d826c794643a1"} Oct 31 12:43:03.670089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.670046 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:03.672542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.672515 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" event={"ID":"aca75aad-681e-48c3-9c57-7f2debcb4925","Type":"ContainerStarted","Data":"52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4"} Oct 31 12:43:03.672658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.672566 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" event={"ID":"aca75aad-681e-48c3-9c57-7f2debcb4925","Type":"ContainerStarted","Data":"826fe5cb02925f723e857f7d0160d3a417870e8f90822bf99e7ed097e9851868"} Oct 31 12:43:03.672658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.672594 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:03.712881 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.712818 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" podStartSLOduration=1.712803011 podStartE2EDuration="1.712803011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:03.693745565 +0000 UTC m=+894.508759962" watchObservedRunningTime="2026-10-31 12:43:03.712803011 +0000 UTC m=+894.527817400" Oct 31 12:43:03.714793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.714752 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" podStartSLOduration=2.714742572 podStartE2EDuration="2.714742572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:03.711303846 +0000 UTC m=+894.526318247" watchObservedRunningTime="2026-10-31 12:43:03.714742572 +0000 UTC m=+894.529756962" Oct 31 12:43:03.831260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:03.831224 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:04.708220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:04.708195 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:05.616512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:05.616481 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:06.351641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:06.351607 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:06.743864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:06.743796 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:07.094294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.094267 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:07.237960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.237926 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:07.633833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.633802 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:43:07.970598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.970506 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl"] Oct 31 12:43:07.975027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.974995 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:07.981271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.981088 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 12:43:07.981271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.981194 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 12:43:07.981271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.981244 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 12:43:07.981705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:07.981679 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mvht9\"" Oct 31 12:43:08.008178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.008147 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:08.016307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.013823 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl"] Oct 31 12:43:08.058986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.058927 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.059420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.059376 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.059603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.059452 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2vc\" (UniqueName: \"kubernetes.io/projected/fee83ef4-beab-4438-80d9-355b9af9e3d2-kube-api-access-8w2vc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.059684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.059600 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.059684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.059652 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.059858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.059835 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee83ef4-beab-4438-80d9-355b9af9e3d2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.060065 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.060041 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.161791 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.161732 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.162038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.161822 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.162038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.161846 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.162038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.161868 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2vc\" (UniqueName: \"kubernetes.io/projected/fee83ef4-beab-4438-80d9-355b9af9e3d2-kube-api-access-8w2vc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.162038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.161951 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.162038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.161989 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.162386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.162180 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee83ef4-beab-4438-80d9-355b9af9e3d2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.162675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.162650 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.164670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.164644 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.164670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.164660 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee83ef4-beab-4438-80d9-355b9af9e3d2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.164966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.164945 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.165069 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.164974 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.177770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.176749 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee83ef4-beab-4438-80d9-355b9af9e3d2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.177770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.177133 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2vc\" (UniqueName: \"kubernetes.io/projected/fee83ef4-beab-4438-80d9-355b9af9e3d2-kube-api-access-8w2vc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ff2sl\" (UID: \"fee83ef4-beab-4438-80d9-355b9af9e3d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.306658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.306589 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:08.593789 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.593763 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:08.623514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.623485 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl"] Oct 31 12:43:08.624894 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:08.624866 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee83ef4_beab_4438_80d9_355b9af9e3d2.slice/crio-4673e84bd0a90accb930c3f0de12ae297cb2196548e08740b351c1d053417f59 WatchSource:0}: Error finding container 4673e84bd0a90accb930c3f0de12ae297cb2196548e08740b351c1d053417f59: Status 404 returned error can't find the container with id 4673e84bd0a90accb930c3f0de12ae297cb2196548e08740b351c1d053417f59 Oct 31 12:43:08.626880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.626861 6727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:43:08.713045 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.713014 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" event={"ID":"fee83ef4-beab-4438-80d9-355b9af9e3d2","Type":"ContainerStarted","Data":"4673e84bd0a90accb930c3f0de12ae297cb2196548e08740b351c1d053417f59"} Oct 31 12:43:08.737085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.737063 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:08.923539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:08.922909 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:09.291989 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:09.291907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:09.666489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:09.660764 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:09.909993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:09.909962 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:10.404512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:10.404441 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:11.340885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:11.340847 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:12.302668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:12.302636 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:12.444944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:12.438827 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:12.656550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:12.656430 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:13.070648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.070612 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:43:13.070797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.070691 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:43:13.097807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.097782 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:13.241520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.241449 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:13.440740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.440708 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:13.677148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.677119 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:13.786539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.785542 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" event={"ID":"fee83ef4-beab-4438-80d9-355b9af9e3d2","Type":"ContainerStarted","Data":"edb5aa00682feb7eec0f5b95f300b6108db81e2f29dffb4d5c68ff4e5509ccb8"} Oct 31 12:43:13.786539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.785629 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:13.803995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.803957 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:13.821311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.821263 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" podStartSLOduration=2.37903992 podStartE2EDuration="6.821249997s" totalImagesPullingTime="4.442210077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:13.817002473 +0000 UTC m=+904.632016863" watchObservedRunningTime="2026-10-31 12:43:13.821249997 +0000 UTC m=+904.636264387" Oct 31 12:43:13.887969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.887930 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc"] Oct 31 12:43:13.895125 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.895096 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:13.910848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.908058 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc"] Oct 31 12:43:13.970536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.963722 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-43c7f8da-dockercfg-85gwm\"" Oct 31 12:43:13.970536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:13.967604 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-43c7f8da\"" Oct 31 12:43:14.013302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.013264 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc796eb-38dd-4420-876c-f827d9af160b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:14.022354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.022318 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgd9\" (UniqueName: \"kubernetes.io/projected/dfc796eb-38dd-4420-876c-f827d9af160b-kube-api-access-dtgd9\") pod \"payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:14.033536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.033300 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:14.161486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.158402 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgd9\" (UniqueName: \"kubernetes.io/projected/dfc796eb-38dd-4420-876c-f827d9af160b-kube-api-access-dtgd9\") pod \"payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:14.161486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.158701 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc796eb-38dd-4420-876c-f827d9af160b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:14.176478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.176065 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc796eb-38dd-4420-876c-f827d9af160b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:14.266538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.247182 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgd9\" (UniqueName: \"kubernetes.io/projected/dfc796eb-38dd-4420-876c-f827d9af160b-kube-api-access-dtgd9\") pod \"payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:14.532201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.532100 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:14.656682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.656606 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:14.710545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.693329 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:14.804493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:14.800591 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ff2sl" Oct 31 12:43:15.042865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:15.042829 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc"] Oct 31 12:43:15.044527 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:15.044497 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc796eb_38dd_4420_876c_f827d9af160b.slice/crio-c9136ee3bd3a18c5c9272dad517260c69eafcb8d482f2ba2d133a7871f3d5dee WatchSource:0}: Error finding container c9136ee3bd3a18c5c9272dad517260c69eafcb8d482f2ba2d133a7871f3d5dee: Status 404 returned error can't find the container with id c9136ee3bd3a18c5c9272dad517260c69eafcb8d482f2ba2d133a7871f3d5dee Oct 31 12:43:15.688992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:15.688967 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:15.689176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:15.689044 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:15.808947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:15.808911 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" event={"ID":"dfc796eb-38dd-4420-876c-f827d9af160b","Type":"ContainerStarted","Data":"858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081"} Oct 31 12:43:15.809414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:15.808982 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" event={"ID":"dfc796eb-38dd-4420-876c-f827d9af160b","Type":"ContainerStarted","Data":"c9136ee3bd3a18c5c9272dad517260c69eafcb8d482f2ba2d133a7871f3d5dee"} Oct 31 12:43:15.809414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:15.809261 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:15.839518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:15.839115 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" podStartSLOduration=2.8390992390000003 podStartE2EDuration="2.839099239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:15.834910508 +0000 UTC m=+906.649924901" watchObservedRunningTime="2026-10-31 12:43:15.839099239 +0000 UTC m=+906.654113631" Oct 31 12:43:16.738360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:16.738327 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:17.811375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:17.811335 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:17.952860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:17.952824 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:18.319183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:18.319130 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:18.504409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:18.504375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:18.853799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:18.853764 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:19.408581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:19.408540 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:19.828403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:19.828372 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:20.707955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:20.707925 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:21.601314 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:21.601272 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:22.451613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:22.451575 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:23.235723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:23.235696 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:23.388980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:23.388942 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:23.708221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:23.708194 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:23.868021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:23.867985 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:24.352279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:24.352211 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:24.942224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:24.942195 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:24.994099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:24.994049 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b"] Oct 31 12:43:24.999119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:24.999097 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.003858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.003835 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ed25a690\"" Oct 31 12:43:25.009344 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.009319 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b"] Oct 31 12:43:25.034669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.034639 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ed25a690-dockercfg-j6lg6\"" Oct 31 12:43:25.069146 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.069015 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wtd\" (UniqueName: \"kubernetes.io/projected/a47a8e05-50ff-4398-b60c-24f78bdc9c84-kube-api-access-h4wtd\") pod \"payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.069530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.069502 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a8e05-50ff-4398-b60c-24f78bdc9c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.091307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.091268 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:25.184486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.181781 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wtd\" (UniqueName: \"kubernetes.io/projected/a47a8e05-50ff-4398-b60c-24f78bdc9c84-kube-api-access-h4wtd\") pod \"payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.184486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.181986 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a8e05-50ff-4398-b60c-24f78bdc9c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.244541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.217915 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a8e05-50ff-4398-b60c-24f78bdc9c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.292486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.270166 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wtd\" (UniqueName: \"kubernetes.io/projected/a47a8e05-50ff-4398-b60c-24f78bdc9c84-kube-api-access-h4wtd\") pod \"payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.330228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.330199 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.568638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.568521 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:25.706936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.706907 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b"] Oct 31 12:43:25.934667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.934633 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" event={"ID":"a47a8e05-50ff-4398-b60c-24f78bdc9c84","Type":"ContainerStarted","Data":"e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d"} Oct 31 12:43:25.934865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.934694 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" event={"ID":"a47a8e05-50ff-4398-b60c-24f78bdc9c84","Type":"ContainerStarted","Data":"620f0e0f6d258a881e048c9c042dc0744640b423de38dc12b18c59c1864e3bbe"} Oct 31 12:43:25.934865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.934727 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:25.954446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:25.954392 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" podStartSLOduration=1.9543737559999999 podStartE2EDuration="1.954373756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:25.952135067 +0000 UTC m=+916.767149457" watchObservedRunningTime="2026-10-31 12:43:25.954373756 +0000 UTC m=+916.769388148" Oct 31 12:43:26.501489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:26.497703 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:26.619586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:26.619548 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v"] Oct 31 12:43:26.777518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:26.777363 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:26.820708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:26.820678 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:26.911873 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:26.911838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:27.534803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:27.534769 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:28.326518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:28.326455 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:28.698911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:28.698837 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:43:29.279572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:29.279542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:30.112148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:30.112115 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:30.345023 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:30.344989 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:43:31.389495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.377677 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:31.654133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.654012 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" podUID="d9627a91-c210-48bc-a39e-40429c01645a" containerName="payload-processing" containerID="cri-o://05343067a8408e6c5e0c74ec5facb75882bc83b3decba772ce6cfdae31f278f2" gracePeriod=25 Oct 31 12:43:31.865489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.865375 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:31.865840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.865756 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:31.909854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.909765 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:31.952197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.952151 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:31.991121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.980572 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:31.991121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:31.981032 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:43:32.095629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.095594 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:43:32.095934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.095887 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.096563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.096529 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:43:32.096831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.096810 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.214567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.202445 6727 generic.go:356] "Generic (PLEG): container finished" podID="d9627a91-c210-48bc-a39e-40429c01645a" containerID="05343067a8408e6c5e0c74ec5facb75882bc83b3decba772ce6cfdae31f278f2" exitCode=0 Oct 31 12:43:32.214567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.202693 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" event={"ID":"d9627a91-c210-48bc-a39e-40429c01645a","Type":"ContainerDied","Data":"05343067a8408e6c5e0c74ec5facb75882bc83b3decba772ce6cfdae31f278f2"} Oct 31 12:43:32.452489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.446228 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.486730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.486638 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.487518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.486946 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.516488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.513733 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.516488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.514105 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:43:32.698488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.683555 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.698488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.683951 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:43:32.780197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.779733 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:32.869486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.866290 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:43:32.869486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.866632 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:43:32.869486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.867235 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:43:32.869486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.867304 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:43:32.919487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:32.911416 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:33.005102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.002885 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:43:33.056753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.002924 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:43:33.116519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.005163 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:43:33.116519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.061870 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:43:33.116519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.061918 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:43:33.116519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.079922 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 31 12:43:33.203580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.203535 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 31 12:43:33.341082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.236564 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 12:43:33.341082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.302042 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:43:33.341082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.335216 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:33.446848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.446793 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:33.501545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.494531 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:43:33.529544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.502135 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:33.557936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.556858 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:33.557936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.557086 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:43:33.949945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:33.949890 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:34.576921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.576872 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:34.577903 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.002940 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:43:34.594130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.003160 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:43:34.600567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.003576 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:43:34.617644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.114158 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:34.620921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.157554 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:34.633006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.216961 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:43:34.650510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.540547 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:43:34.790494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.702288 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:43:34.909607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.888064 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" podUID="3b468147-6eba-4cbd-aff5-d13da0f35497" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:35.090661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.891577 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:35.101043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.977040 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 31 12:43:35.165545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.977163 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:43:35.165545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:34.977267 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:43:35.165545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.004908 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:35.251365 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:35.251324 6727 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.923s" Oct 31 12:43:35.251609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.251539 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:43:35.251695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.251616 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:43:35.251695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.251688 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:43:35.251791 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.251749 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:43:35.251840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.251800 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:43:35.251893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.251857 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:43:35.251954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.251917 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:43:35.254884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.254817 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:35.259566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.259543 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:43:35.582487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.578890 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:35.582487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.579024 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:43:35.582487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.579079 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:43:35.901543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.881793 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:43:35.946625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.946570 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:43:36.001509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.976202 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:36.027634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.027595 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:43:36.030126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:35.989556 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:43:36.032588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.030332 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:43:36.034521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.030759 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:43:36.127668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.127612 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:43:36.153491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.153384 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:43:36.172837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.161316 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9627a91-c210-48bc-a39e-40429c01645a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9627a91-c210-48bc-a39e-40429c01645a-plugins-config-volume\") pod \"d9627a91-c210-48bc-a39e-40429c01645a\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " Oct 31 12:43:36.178668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.178628 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9627a91-c210-48bc-a39e-40429c01645a-kube-api-access-99dds\" (UniqueName: \"kubernetes.io/projected/d9627a91-c210-48bc-a39e-40429c01645a-kube-api-access-99dds\") pod \"d9627a91-c210-48bc-a39e-40429c01645a\" (UID: \"d9627a91-c210-48bc-a39e-40429c01645a\") " Oct 31 12:43:36.412537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.162571 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:36.447869 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.442765 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9627a91-c210-48bc-a39e-40429c01645a-kube-api-access-99dds" pod "d9627a91-c210-48bc-a39e-40429c01645a" (UID: "d9627a91-c210-48bc-a39e-40429c01645a"). InnerVolumeSpecName "kube-api-access-99dds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:36.479071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.472219 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dds\" (UniqueName: \"kubernetes.io/projected/d9627a91-c210-48bc-a39e-40429c01645a-kube-api-access-99dds\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:36.556564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.547729 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9627a91-c210-48bc-a39e-40429c01645a-plugins-config-volume" pod "d9627a91-c210-48bc-a39e-40429c01645a" (UID: "d9627a91-c210-48bc-a39e-40429c01645a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:43:36.630509 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:36.569506 6727 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.274s" Oct 31 12:43:36.643609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.643577 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:43:36.700561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.691989 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:36.736264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.736209 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9627a91-c210-48bc-a39e-40429c01645a-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:36.867206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.826457 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" event={"ID":"d9627a91-c210-48bc-a39e-40429c01645a","Type":"ContainerDied","Data":"1bba515d9b490ee98c09132e3dacac2781fc71a4e6fe26b1953ecb97ea45b6aa"} Oct 31 12:43:36.867206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.826543 6727 scope.go:122] "RemoveContainer" containerID="05343067a8408e6c5e0c74ec5facb75882bc83b3decba772ce6cfdae31f278f2" Oct 31 12:43:36.867206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.826545 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:36.867206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.826731 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v" Oct 31 12:43:36.867206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.843518 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:43:36.867206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.843776 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:43:36.888488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.872586 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn"] Oct 31 12:43:36.955990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.955905 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:36.959132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:36.959052 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:37.024714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.024641 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:43:37.028006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.027957 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:43:37.058609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.044610 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:37.058609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.045690 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:37.065203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.065173 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7"] Oct 31 12:43:37.093600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.093559 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:43:37.118255 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.102626 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:37.130158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.130129 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:43:37.130601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.130580 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:43:37.143912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.143889 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:43:37.161866 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.161834 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:37.182512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.162978 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:43:37.189737 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.189709 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:43:37.301691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.301663 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:43:37.303363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.303206 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:43:37.317438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.316897 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:37.339892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.339864 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:43:37.340037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.339904 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw"] Oct 31 12:43:37.340037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.339932 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj"] Oct 31 12:43:37.340037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.339964 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:43:37.340037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.339987 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:43:37.340037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.340005 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:43:37.641282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.641248 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v"] Oct 31 12:43:37.646960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.646935 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-59fd8a28-5c764596f-lvf2v"] Oct 31 12:43:37.792488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:37.790691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:38.756210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.756173 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb"] Oct 31 12:43:38.757083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.757056 6727 cpu_manager.go:414] "containerMap: removing container" podUID="d9627a91-c210-48bc-a39e-40429c01645a" containerName="payload-processing" Oct 31 12:43:38.757083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.757084 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9627a91-c210-48bc-a39e-40429c01645a" containerName="payload-processing" Oct 31 12:43:38.757265 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.757221 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9627a91-c210-48bc-a39e-40429c01645a" containerName="payload-processing" Oct 31 12:43:38.785728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.785695 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:38.787524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.787498 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb"] Oct 31 12:43:38.791966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.791355 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ed25a690-serving-cert\"" Oct 31 12:43:38.837910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.837882 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:38.907484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.905106 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:38.907484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.905214 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-openshift-service-ca\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:38.907484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.905284 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-metrics-certs\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:38.907484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.905634 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwkq\" (UniqueName: \"kubernetes.io/projected/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-kube-api-access-4mwkq\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:38.976365 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:38.976319 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:39.007241 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.007160 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwkq\" (UniqueName: \"kubernetes.io/projected/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-kube-api-access-4mwkq\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.007395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.007258 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.007395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.007333 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-openshift-service-ca\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.007395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.007389 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-metrics-certs\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.007604 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:39.007421 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-serving-cert: secret "maas-api-e2e-mt-ed25a690-serving-cert" not found Oct 31 12:43:39.007604 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:39.007507 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls podName:36b7dc9f-c8e0-4e5b-8207-fe52c90d0352 nodeName:}" failed. No retries permitted until 2026-10-31 12:43:39.507488246 +0000 UTC m=+930.322502633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls") pod "maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" (UID: "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352") : secret "maas-api-e2e-mt-ed25a690-serving-cert" not found Oct 31 12:43:39.008053 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.008031 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-openshift-service-ca\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.009843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.009823 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-metrics-certs\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.021322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.021295 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwkq\" (UniqueName: \"kubernetes.io/projected/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-kube-api-access-4mwkq\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.302834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.302743 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9627a91-c210-48bc-a39e-40429c01645a" path="/var/lib/kubelet/pods/d9627a91-c210-48bc-a39e-40429c01645a/volumes" Oct 31 12:43:39.518978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.518931 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:39.519564 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:39.519539 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-serving-cert: secret "maas-api-e2e-mt-ed25a690-serving-cert" not found Oct 31 12:43:39.519690 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:39.519627 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls podName:36b7dc9f-c8e0-4e5b-8207-fe52c90d0352 nodeName:}" failed. No retries permitted until 2026-10-31 12:43:40.519600075 +0000 UTC m=+931.334614505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls") pod "maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" (UID: "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352") : secret "maas-api-e2e-mt-ed25a690-serving-cert" not found Oct 31 12:43:39.896707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:39.876660 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:40.538296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:40.538258 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:40.538458 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:40.538414 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-serving-cert: secret "maas-api-e2e-mt-ed25a690-serving-cert" not found Oct 31 12:43:40.538545 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:40.538509 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls podName:36b7dc9f-c8e0-4e5b-8207-fe52c90d0352 nodeName:}" failed. No retries permitted until 2026-10-31 12:43:42.53849153 +0000 UTC m=+933.353505913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls") pod "maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" (UID: "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352") : secret "maas-api-e2e-mt-ed25a690-serving-cert" not found Oct 31 12:43:41.001030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:41.000987 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:41.526301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:41.526267 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:41.661282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:41.661252 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:41.826547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:41.822041 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:41.935528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:41.934113 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" podUID="d87dc592-2996-452d-b4de-6311df85a230" containerName="payload-processing" containerID="cri-o://6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a" gracePeriod=25 Oct 31 12:43:42.004111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.004080 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:42.066353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.066315 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" podUID="cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" containerName="payload-pre-processing" containerID="cri-o://9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f" gracePeriod=25 Oct 31 12:43:42.154845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.154813 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:42.343920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.343831 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" podUID="aca75aad-681e-48c3-9c57-7f2debcb4925" containerName="payload-pre-processing" containerID="cri-o://52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4" gracePeriod=25 Oct 31 12:43:42.344346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.343848 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" podUID="bd0f1404-c7a4-441e-884f-6559609be9aa" containerName="payload-processing" containerID="cri-o://5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26" gracePeriod=25 Oct 31 12:43:42.535185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.535008 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:42.545550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.540649 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:42.565657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.565620 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:42.575872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.575842 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") pod \"maas-api-e2e-mt-ed25a690-55c58cff5b-65khb\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:42.663669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.663641 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:42.667070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.667049 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-kube-api-access-j5xsj\" (UniqueName: \"kubernetes.io/projected/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-kube-api-access-j5xsj\") pod \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " Oct 31 12:43:42.667588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.667571 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87dc592-2996-452d-b4de-6311df85a230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87dc592-2996-452d-b4de-6311df85a230-plugins-config-volume\") pod \"d87dc592-2996-452d-b4de-6311df85a230\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " Oct 31 12:43:42.667823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.667812 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-plugins-config-volume\") pod \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\" (UID: \"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e\") " Oct 31 12:43:42.667942 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.667925 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87dc592-2996-452d-b4de-6311df85a230-kube-api-access-4577g\" (UniqueName: \"kubernetes.io/projected/d87dc592-2996-452d-b4de-6311df85a230-kube-api-access-4577g\") pod \"d87dc592-2996-452d-b4de-6311df85a230\" (UID: \"d87dc592-2996-452d-b4de-6311df85a230\") " Oct 31 12:43:42.668231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.668207 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87dc592-2996-452d-b4de-6311df85a230-plugins-config-volume" pod "d87dc592-2996-452d-b4de-6311df85a230" (UID: "d87dc592-2996-452d-b4de-6311df85a230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:43:42.668626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.668603 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-plugins-config-volume" pod "cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" (UID: "cf5694ad-aec8-4ad5-8ce5-d93e42becd6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:43:42.669022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.669002 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87dc592-2996-452d-b4de-6311df85a230-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.669142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.669127 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.673648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.673615 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-kube-api-access-j5xsj" pod "cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" (UID: "cf5694ad-aec8-4ad5-8ce5-d93e42becd6e"). InnerVolumeSpecName "kube-api-access-j5xsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:42.674982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.674945 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87dc592-2996-452d-b4de-6311df85a230-kube-api-access-4577g" pod "d87dc592-2996-452d-b4de-6311df85a230" (UID: "d87dc592-2996-452d-b4de-6311df85a230"). InnerVolumeSpecName "kube-api-access-4577g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:42.718595 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.718551 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:42.736186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.736163 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:42.770710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.770672 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca75aad-681e-48c3-9c57-7f2debcb4925-kube-api-access-bdfxf\" (UniqueName: \"kubernetes.io/projected/aca75aad-681e-48c3-9c57-7f2debcb4925-kube-api-access-bdfxf\") pod \"aca75aad-681e-48c3-9c57-7f2debcb4925\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " Oct 31 12:43:42.770874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.770858 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca75aad-681e-48c3-9c57-7f2debcb4925-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca75aad-681e-48c3-9c57-7f2debcb4925-plugins-config-volume\") pod \"aca75aad-681e-48c3-9c57-7f2debcb4925\" (UID: \"aca75aad-681e-48c3-9c57-7f2debcb4925\") " Oct 31 12:43:42.771318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.771287 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca75aad-681e-48c3-9c57-7f2debcb4925-plugins-config-volume" pod "aca75aad-681e-48c3-9c57-7f2debcb4925" (UID: "aca75aad-681e-48c3-9c57-7f2debcb4925"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:43:42.771700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.771678 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4577g\" (UniqueName: \"kubernetes.io/projected/d87dc592-2996-452d-b4de-6311df85a230-kube-api-access-4577g\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.771700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.771701 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca75aad-681e-48c3-9c57-7f2debcb4925-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.771835 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.771711 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5xsj\" (UniqueName: \"kubernetes.io/projected/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e-kube-api-access-j5xsj\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.773611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.773571 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca75aad-681e-48c3-9c57-7f2debcb4925-kube-api-access-bdfxf" pod "aca75aad-681e-48c3-9c57-7f2debcb4925" (UID: "aca75aad-681e-48c3-9c57-7f2debcb4925"). InnerVolumeSpecName "kube-api-access-bdfxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:42.872641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.872552 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0f1404-c7a4-441e-884f-6559609be9aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f1404-c7a4-441e-884f-6559609be9aa-plugins-config-volume\") pod \"bd0f1404-c7a4-441e-884f-6559609be9aa\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " Oct 31 12:43:42.872911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.872890 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0f1404-c7a4-441e-884f-6559609be9aa-kube-api-access-9mjnt\" (UniqueName: \"kubernetes.io/projected/bd0f1404-c7a4-441e-884f-6559609be9aa-kube-api-access-9mjnt\") pod \"bd0f1404-c7a4-441e-884f-6559609be9aa\" (UID: \"bd0f1404-c7a4-441e-884f-6559609be9aa\") " Oct 31 12:43:42.873724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.873699 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdfxf\" (UniqueName: \"kubernetes.io/projected/aca75aad-681e-48c3-9c57-7f2debcb4925-kube-api-access-bdfxf\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.874624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.874581 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0f1404-c7a4-441e-884f-6559609be9aa-plugins-config-volume" pod "bd0f1404-c7a4-441e-884f-6559609be9aa" (UID: "bd0f1404-c7a4-441e-884f-6559609be9aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:43:42.876018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.875996 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0f1404-c7a4-441e-884f-6559609be9aa-kube-api-access-9mjnt" pod "bd0f1404-c7a4-441e-884f-6559609be9aa" (UID: "bd0f1404-c7a4-441e-884f-6559609be9aa"). InnerVolumeSpecName "kube-api-access-9mjnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:42.975246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.975205 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjnt\" (UniqueName: \"kubernetes.io/projected/bd0f1404-c7a4-441e-884f-6559609be9aa-kube-api-access-9mjnt\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.975246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.975244 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd0f1404-c7a4-441e-884f-6559609be9aa-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:42.981824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.981795 6727 generic.go:356] "Generic (PLEG): container finished" podID="bd0f1404-c7a4-441e-884f-6559609be9aa" containerID="5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26" exitCode=0 Oct 31 12:43:42.981975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.981843 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" Oct 31 12:43:42.981975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.981886 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" event={"ID":"bd0f1404-c7a4-441e-884f-6559609be9aa","Type":"ContainerDied","Data":"5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26"} Oct 31 12:43:42.982101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.981993 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw" event={"ID":"bd0f1404-c7a4-441e-884f-6559609be9aa","Type":"ContainerDied","Data":"0e954f417aeb354bee27cde39e63e6696478332e3ed37649094d826c794643a1"} Oct 31 12:43:42.982101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.982014 6727 scope.go:122] "RemoveContainer" containerID="5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26" Oct 31 12:43:42.984572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.984407 6727 generic.go:356] "Generic (PLEG): container finished" podID="aca75aad-681e-48c3-9c57-7f2debcb4925" containerID="52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4" exitCode=0 Oct 31 12:43:42.984572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.984484 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" Oct 31 12:43:42.984572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.984520 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" event={"ID":"aca75aad-681e-48c3-9c57-7f2debcb4925","Type":"ContainerDied","Data":"52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4"} Oct 31 12:43:42.984731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.984565 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj" event={"ID":"aca75aad-681e-48c3-9c57-7f2debcb4925","Type":"ContainerDied","Data":"826fe5cb02925f723e857f7d0160d3a417870e8f90822bf99e7ed097e9851868"} Oct 31 12:43:42.986156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.986129 6727 generic.go:356] "Generic (PLEG): container finished" podID="d87dc592-2996-452d-b4de-6311df85a230" containerID="6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a" exitCode=0 Oct 31 12:43:42.986279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.986187 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" event={"ID":"d87dc592-2996-452d-b4de-6311df85a230","Type":"ContainerDied","Data":"6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a"} Oct 31 12:43:42.986279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.986248 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" event={"ID":"d87dc592-2996-452d-b4de-6311df85a230","Type":"ContainerDied","Data":"a4b49eb649831214c1da0de2d8c08407a165fdc6221fccc92dfc1f98a704910f"} Oct 31 12:43:42.986392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.986330 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn" Oct 31 12:43:42.991880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.991691 6727 generic.go:356] "Generic (PLEG): container finished" podID="cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" containerID="9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f" exitCode=0 Oct 31 12:43:42.991880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.991764 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" Oct 31 12:43:42.991880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.991818 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" event={"ID":"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e","Type":"ContainerDied","Data":"9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f"} Oct 31 12:43:42.991880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:42.991851 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7" event={"ID":"cf5694ad-aec8-4ad5-8ce5-d93e42becd6e","Type":"ContainerDied","Data":"d71eb6915e715b128ac6e2f44ff15efee2cac003cac9080036a1ddff5419f982"} Oct 31 12:43:43.025652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.025421 6727 scope.go:122] "RemoveContainer" containerID="5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26" Oct 31 12:43:43.025990 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:43.025947 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26\": container with ID starting with 5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26 not found: ID does not exist" containerID="5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26" Oct 31 12:43:43.026080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.026008 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26"} err="failed to get container status \"5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26\": rpc error: code = NotFound desc = could not find container \"5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26\": container with ID starting with 5512a90b1552ce82d42830586d0884916ae90b24de229c5fb507a6485d910a26 not found: ID does not exist" Oct 31 12:43:43.026080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.026039 6727 scope.go:122] "RemoveContainer" containerID="52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4" Oct 31 12:43:43.026271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.026251 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb"] Oct 31 12:43:43.060407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.060380 6727 scope.go:122] "RemoveContainer" containerID="52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4" Oct 31 12:43:43.060841 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:43.060786 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4\": container with ID starting with 52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4 not found: ID does not exist" containerID="52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4" Oct 31 12:43:43.060949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.060855 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4"} err="failed to get container status \"52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4\": rpc error: code = NotFound desc = could not find container \"52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4\": container with ID starting with 52f562260cf10f83ba3ee3f17748a94113cea631b62e209fe6038b61aeadfdb4 not found: ID does not exist" Oct 31 12:43:43.060949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.060881 6727 scope.go:122] "RemoveContainer" containerID="6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a" Oct 31 12:43:43.068931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.068898 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw"] Oct 31 12:43:43.073253 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.073228 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b849a4c4-6c6587f68f-cxmmw"] Oct 31 12:43:43.079493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.079377 6727 scope.go:122] "RemoveContainer" containerID="6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a" Oct 31 12:43:43.079764 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:43.079740 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a\": container with ID starting with 6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a not found: ID does not exist" containerID="6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a" Oct 31 12:43:43.079830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.079777 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a"} err="failed to get container status \"6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a\": rpc error: code = NotFound desc = could not find container \"6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a\": container with ID starting with 6a3d9a7545d3b8b75841fdd3aca047b784c860225dafca2cc0404700dce89d0a not found: ID does not exist" Oct 31 12:43:43.079830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.079801 6727 scope.go:122] "RemoveContainer" containerID="9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f" Oct 31 12:43:43.082860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.082838 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj"] Oct 31 12:43:43.091044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.091020 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b849a4c4-7698d95fd7-4ghrj"] Oct 31 12:43:43.093018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.093000 6727 scope.go:122] "RemoveContainer" containerID="9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f" Oct 31 12:43:43.093321 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:43.093299 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f\": container with ID starting with 9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f not found: ID does not exist" containerID="9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f" Oct 31 12:43:43.093383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.093329 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f"} err="failed to get container status \"9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f\": rpc error: code = NotFound desc = could not find container \"9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f\": container with ID starting with 9688d0176c342c8af63891be13aa67685facb64813a42fd691a037db3104935f not found: ID does not exist" Oct 31 12:43:43.103536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.103508 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn"] Oct 31 12:43:43.108373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.108350 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5e8c3f2-7454c78cc9-s6srn"] Oct 31 12:43:43.110387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.110366 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:43.119644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.119622 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7"] Oct 31 12:43:43.125038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.124974 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5e8c3f2-f4cf99959-jxdb7"] Oct 31 12:43:43.307819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.307781 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca75aad-681e-48c3-9c57-7f2debcb4925" path="/var/lib/kubelet/pods/aca75aad-681e-48c3-9c57-7f2debcb4925/volumes" Oct 31 12:43:43.308521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.308497 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0f1404-c7a4-441e-884f-6559609be9aa" path="/var/lib/kubelet/pods/bd0f1404-c7a4-441e-884f-6559609be9aa/volumes" Oct 31 12:43:43.309099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.309079 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" path="/var/lib/kubelet/pods/cf5694ad-aec8-4ad5-8ce5-d93e42becd6e/volumes" Oct 31 12:43:43.309662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.309641 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87dc592-2996-452d-b4de-6311df85a230" path="/var/lib/kubelet/pods/d87dc592-2996-452d-b4de-6311df85a230/volumes" Oct 31 12:43:43.681257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.681220 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz"] Oct 31 12:43:43.681552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.681456 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" podUID="afd869ac-e11b-44fd-b55e-90d4e7a591d6" containerName="istio-proxy" containerID="cri-o://944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0" gracePeriod=30 Oct 31 12:43:43.841932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:43.837780 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:44.002788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.002711 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:44.043506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.043450 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" event={"ID":"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352","Type":"ContainerStarted","Data":"7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2"} Oct 31 12:43:44.043955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.043535 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" event={"ID":"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352","Type":"ContainerStarted","Data":"7774183d67a413914b4f7406058862b6cc0ec40abd06f3183dfc799cf43bf069"} Oct 31 12:43:44.044627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.044601 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:44.068742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.068691 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" podStartSLOduration=5.652415964 podStartE2EDuration="6.068676397s" totalImagesPullingTime="416.260433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:44.065684226 +0000 UTC m=+934.880698616" watchObservedRunningTime="2026-10-31 12:43:44.068676397 +0000 UTC m=+934.883690787" Oct 31 12:43:44.195292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.195209 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:44.366840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.366806 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:44.758329 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.758294 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw"] Oct 31 12:43:44.759911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.759879 6727 cpu_manager.go:414] "containerMap: removing container" podUID="d87dc592-2996-452d-b4de-6311df85a230" containerName="payload-processing" Oct 31 12:43:44.759911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.759911 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87dc592-2996-452d-b4de-6311df85a230" containerName="payload-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.759942 6727 cpu_manager.go:414] "containerMap: removing container" podUID="bd0f1404-c7a4-441e-884f-6559609be9aa" containerName="payload-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.759951 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0f1404-c7a4-441e-884f-6559609be9aa" containerName="payload-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.759977 6727 cpu_manager.go:414] "containerMap: removing container" podUID="cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" containerName="payload-pre-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.759984 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" containerName="payload-pre-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.760004 6727 cpu_manager.go:414] "containerMap: removing container" podUID="aca75aad-681e-48c3-9c57-7f2debcb4925" containerName="payload-pre-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.760011 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca75aad-681e-48c3-9c57-7f2debcb4925" containerName="payload-pre-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.760189 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87dc592-2996-452d-b4de-6311df85a230" containerName="payload-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.760207 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0f1404-c7a4-441e-884f-6559609be9aa" containerName="payload-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.760222 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5694ad-aec8-4ad5-8ce5-d93e42becd6e" containerName="payload-pre-processing" Oct 31 12:43:44.760243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.760234 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca75aad-681e-48c3-9c57-7f2debcb4925" containerName="payload-pre-processing" Oct 31 12:43:44.765952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.764211 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.768787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.768763 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw"] Oct 31 12:43:44.769354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.769335 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d702046e-openshift-default-dockercfg-799nm\"" Oct 31 12:43:44.830303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.830226 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878345 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-envoy\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878422 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-socket\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878491 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1906916a-b650-415c-b664-80303d614c70-istiod-ca-cert\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878585 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-credential-socket\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878610 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-data\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878633 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-istio-token\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878765 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1906916a-b650-415c-b664-80303d614c70-istio-podinfo\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.878873 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-certs\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.896521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.879093 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975xv\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-kube-api-access-975xv\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981416 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975xv\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-kube-api-access-975xv\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981491 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-envoy\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981539 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-socket\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981585 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1906916a-b650-415c-b664-80303d614c70-istiod-ca-cert\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981672 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-credential-socket\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981698 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-data\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981723 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-istio-token\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981855 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1906916a-b650-415c-b664-80303d614c70-istio-podinfo\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.981951 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-certs\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.983772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.983121 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1906916a-b650-415c-b664-80303d614c70-istiod-ca-cert\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.985484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.984655 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-data\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.985484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.984890 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-credential-socket\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.986035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.985706 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-certs\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:44.986035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:44.985990 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-socket\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:45.003313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.003222 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1906916a-b650-415c-b664-80303d614c70-istio-podinfo\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:45.050564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.021969 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-istio-token\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:45.050564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.022641 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975xv\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-kube-api-access-975xv\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:45.068486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.057669 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-envoy\") pod \"e2e-ait-d702046e-openshift-default-fb99775bf-wljxw\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:45.093507 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.089336 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:45.094356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.094326 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:45.234257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.233966 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:45.455766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.455738 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw"] Oct 31 12:43:45.455886 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:45.455786 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1906916a_b650_415c_b664_80303d614c70.slice/crio-5a4ddf39986ada3b2cf444b7d29d19ad61ecb1d49b79375548ed2e78039230ae WatchSource:0}: Error finding container 5a4ddf39986ada3b2cf444b7d29d19ad61ecb1d49b79375548ed2e78039230ae: Status 404 returned error can't find the container with id 5a4ddf39986ada3b2cf444b7d29d19ad61ecb1d49b79375548ed2e78039230ae Oct 31 12:43:45.459761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.459728 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:43:45.459854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.459797 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:43:45.459854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.459828 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:43:45.797795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:45.797716 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:46.105704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.105672 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" event={"ID":"1906916a-b650-415c-b664-80303d614c70","Type":"ContainerStarted","Data":"85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e"} Oct 31 12:43:46.108331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.105728 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" event={"ID":"1906916a-b650-415c-b664-80303d614c70","Type":"ContainerStarted","Data":"5a4ddf39986ada3b2cf444b7d29d19ad61ecb1d49b79375548ed2e78039230ae"} Oct 31 12:43:46.138700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.138626 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" podStartSLOduration=2.138606509 podStartE2EDuration="2.138606509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:46.129543505 +0000 UTC m=+936.944557895" watchObservedRunningTime="2026-10-31 12:43:46.138606509 +0000 UTC m=+936.953620901" Oct 31 12:43:46.211848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.211823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:46.371024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.370940 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:46.483491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.476534 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.561561 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-kube-api-access-54glf\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-kube-api-access-54glf\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.561693 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-token\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-token\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.561745 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-podinfo\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.561808 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-envoy\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.562037 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-socket\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.562093 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-certs\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.562119 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-credential-socket\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.562153 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istiod-ca-cert\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.563522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.562193 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-data\") pod \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\" (UID: \"afd869ac-e11b-44fd-b55e-90d4e7a591d6\") " Oct 31 12:43:46.569025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.567781 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-socket" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:43:46.569025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.567997 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-certs" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:43:46.569025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.568254 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-credential-socket" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:43:46.573493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.572985 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istiod-ca-cert" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:43:46.573637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.573575 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:46.598304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.578623 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-data" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:43:46.598304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.591886 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-envoy" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:43:46.609770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.601572 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-podinfo" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:43:46.627551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.621977 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-kube-api-access-54glf" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "kube-api-access-54glf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:46.627551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.622954 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-token" pod "afd869ac-e11b-44fd-b55e-90d4e7a591d6" (UID: "afd869ac-e11b-44fd-b55e-90d4e7a591d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686316 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686343 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686352 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686361 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686370 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686377 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686385 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686393 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd869ac-e11b-44fd-b55e-90d4e7a591d6-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.686423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.686402 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54glf\" (UniqueName: \"kubernetes.io/projected/afd869ac-e11b-44fd-b55e-90d4e7a591d6-kube-api-access-54glf\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:46.756910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:46.756797 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:47.091742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.091688 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:47.097352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.097325 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:47.117509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.114953 6727 generic.go:356] "Generic (PLEG): container finished" podID="afd869ac-e11b-44fd-b55e-90d4e7a591d6" containerID="944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0" exitCode=0 Oct 31 12:43:47.117509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.114997 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" event={"ID":"afd869ac-e11b-44fd-b55e-90d4e7a591d6","Type":"ContainerDied","Data":"944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0"} Oct 31 12:43:47.117509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.115067 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" event={"ID":"afd869ac-e11b-44fd-b55e-90d4e7a591d6","Type":"ContainerDied","Data":"f3460beddb1716a800f302f454ba2c2b14de9b5646e4d0d1f7acb15cb12300d1"} Oct 31 12:43:47.117509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.115090 6727 scope.go:122] "RemoveContainer" containerID="944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0" Oct 31 12:43:47.117509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.115007 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz" Oct 31 12:43:47.117509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.115514 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:47.117509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.116428 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:43:47.127375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.127339 6727 scope.go:122] "RemoveContainer" containerID="944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0" Oct 31 12:43:47.127677 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:47.127654 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0\": container with ID starting with 944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0 not found: ID does not exist" containerID="944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0" Oct 31 12:43:47.127782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.127686 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0"} err="failed to get container status \"944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0\": rpc error: code = NotFound desc = could not find container \"944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0\": container with ID starting with 944ae407babb20f6375b62c51c7b636317e97f90a2651843d148f3ccd2fb95c0 not found: ID does not exist" Oct 31 12:43:47.204256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.204226 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz"] Oct 31 12:43:47.209143 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.209115 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-59fd8a28-openshift-default-74766997b7-6fxrz"] Oct 31 12:43:47.335191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.335152 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd869ac-e11b-44fd-b55e-90d4e7a591d6" path="/var/lib/kubelet/pods/afd869ac-e11b-44fd-b55e-90d4e7a591d6/volumes" Oct 31 12:43:47.515782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.515685 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:47.692110 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.692030 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:47.847504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:47.845119 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:48.044980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:48.044892 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:48.239000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:48.238962 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:49.029780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:49.029744 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:49.184831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:49.184797 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:49.454325 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:49.454292 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:49.636705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:49.636678 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:49.982621 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:49.982576 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:50.117103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.117074 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx"] Oct 31 12:43:50.117800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.117785 6727 cpu_manager.go:414] "containerMap: removing container" podUID="afd869ac-e11b-44fd-b55e-90d4e7a591d6" containerName="istio-proxy" Oct 31 12:43:50.117863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.117801 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd869ac-e11b-44fd-b55e-90d4e7a591d6" containerName="istio-proxy" Oct 31 12:43:50.117924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.117915 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd869ac-e11b-44fd-b55e-90d4e7a591d6" containerName="istio-proxy" Oct 31 12:43:50.122967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.122947 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.133962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.133930 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d702046e\"" Oct 31 12:43:50.134758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.134737 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d702046e-dockercfg-tgkj7\"" Oct 31 12:43:50.139830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.139803 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx"] Oct 31 12:43:50.182908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.182871 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fd6fca-fc28-4180-8ef5-18454fb4265b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.183053 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.182952 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmhx\" (UniqueName: \"kubernetes.io/projected/14fd6fca-fc28-4180-8ef5-18454fb4265b-kube-api-access-mzmhx\") pod \"payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.192016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.191984 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:50.289537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.285022 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fd6fca-fc28-4180-8ef5-18454fb4265b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.289537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.285070 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmhx\" (UniqueName: \"kubernetes.io/projected/14fd6fca-fc28-4180-8ef5-18454fb4265b-kube-api-access-mzmhx\") pod \"payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.357485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.347796 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmhx\" (UniqueName: \"kubernetes.io/projected/14fd6fca-fc28-4180-8ef5-18454fb4265b-kube-api-access-mzmhx\") pod \"payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.376484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.374054 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fd6fca-fc28-4180-8ef5-18454fb4265b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.410503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.402553 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m"] Oct 31 12:43:50.427121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.426746 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.460139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.460027 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:50.517727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.499345 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m"] Oct 31 12:43:50.597531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.593878 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4nl\" (UniqueName: \"kubernetes.io/projected/83258142-61b9-4760-a622-59ea097fd738-kube-api-access-rg4nl\") pod \"payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.597531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.594096 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83258142-61b9-4760-a622-59ea097fd738-plugins-config-volume\") pod \"payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.695501 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.695451 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4nl\" (UniqueName: \"kubernetes.io/projected/83258142-61b9-4760-a622-59ea097fd738-kube-api-access-rg4nl\") pod \"payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.695688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.695670 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83258142-61b9-4760-a622-59ea097fd738-plugins-config-volume\") pod \"payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.696386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.696368 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83258142-61b9-4760-a622-59ea097fd738-plugins-config-volume\") pod \"payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.707912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.707875 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4nl\" (UniqueName: \"kubernetes.io/projected/83258142-61b9-4760-a622-59ea097fd738-kube-api-access-rg4nl\") pod \"payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.805306 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.805273 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx"] Oct 31 12:43:50.805430 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:50.805336 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fd6fca_fc28_4180_8ef5_18454fb4265b.slice/crio-f669be9a29ea80b35cdcb6cd32afdc35f602f6ac2fc91c77a41c571ee666d0f4 WatchSource:0}: Error finding container f669be9a29ea80b35cdcb6cd32afdc35f602f6ac2fc91c77a41c571ee666d0f4: Status 404 returned error can't find the container with id f669be9a29ea80b35cdcb6cd32afdc35f602f6ac2fc91c77a41c571ee666d0f4 Oct 31 12:43:50.824966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.824941 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:50.842921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:50.842893 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:51.111651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.111561 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:43:51.138304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.137449 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m"] Oct 31 12:43:51.138304 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:51.137611 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83258142_61b9_4760_a622_59ea097fd738.slice/crio-064e7edce5cc4a48a43592ec051fdc8de4eae3e13d063d951da80798bc741973 WatchSource:0}: Error finding container 064e7edce5cc4a48a43592ec051fdc8de4eae3e13d063d951da80798bc741973: Status 404 returned error can't find the container with id 064e7edce5cc4a48a43592ec051fdc8de4eae3e13d063d951da80798bc741973 Oct 31 12:43:51.165671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.165638 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" event={"ID":"14fd6fca-fc28-4180-8ef5-18454fb4265b","Type":"ContainerStarted","Data":"32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16"} Oct 31 12:43:51.165787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.165705 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" event={"ID":"14fd6fca-fc28-4180-8ef5-18454fb4265b","Type":"ContainerStarted","Data":"f669be9a29ea80b35cdcb6cd32afdc35f602f6ac2fc91c77a41c571ee666d0f4"} Oct 31 12:43:51.166810 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.166788 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:43:51.169598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.169569 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" event={"ID":"83258142-61b9-4760-a622-59ea097fd738","Type":"ContainerStarted","Data":"064e7edce5cc4a48a43592ec051fdc8de4eae3e13d063d951da80798bc741973"} Oct 31 12:43:51.192756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.192706 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" podStartSLOduration=1.192686706 podStartE2EDuration="1.192686706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:51.191692984 +0000 UTC m=+942.006707376" watchObservedRunningTime="2026-10-31 12:43:51.192686706 +0000 UTC m=+942.007701098" Oct 31 12:43:51.863754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:51.863718 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:52.185416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:52.185240 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" event={"ID":"83258142-61b9-4760-a622-59ea097fd738","Type":"ContainerStarted","Data":"199a1c06cdab33e7ef9f71198561309d3622429b523b1c3510515f663a566785"} Oct 31 12:43:52.208384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:52.208337 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" podStartSLOduration=2.208320228 podStartE2EDuration="2.208320228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:52.206794182 +0000 UTC m=+943.021808574" watchObservedRunningTime="2026-10-31 12:43:52.208320228 +0000 UTC m=+943.023334623" Oct 31 12:43:52.745724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:52.745690 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:53.193446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:53.193414 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:43:53.531112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:53.531027 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc"] Oct 31 12:43:53.629379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:53.629341 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:53.761012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:53.760947 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:54.064561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:54.064524 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:54.229828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:54.229796 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:54.378669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:54.378642 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:54.668030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:54.667907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:54.885599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:54.885558 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:55.059227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.059112 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:55.520662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.520629 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:55.804406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.804330 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:55.898547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.898519 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv"] Oct 31 12:43:55.903447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.903423 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9"] Oct 31 12:43:55.903634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.903553 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:55.908880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.908861 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:55.915318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.915297 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-9b17b9\"" Oct 31 12:43:55.915690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.915670 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-9b17b9-dockercfg-8l9bd\"" Oct 31 12:43:55.921311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.921288 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv"] Oct 31 12:43:55.942553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.942523 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9"] Oct 31 12:43:55.978694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:55.978667 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:56.004016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.003901 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lr2m\" (UniqueName: \"kubernetes.io/projected/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-kube-api-access-4lr2m\") pod \"payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:56.004181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.004044 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d707124-abc0-4daa-8a91-0af9e6081951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:56.004299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.004258 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:56.004456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.004442 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklsn\" (UniqueName: \"kubernetes.io/projected/1d707124-abc0-4daa-8a91-0af9e6081951-kube-api-access-hklsn\") pod \"payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:56.106682 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.106601 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:56.107054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.107027 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklsn\" (UniqueName: \"kubernetes.io/projected/1d707124-abc0-4daa-8a91-0af9e6081951-kube-api-access-hklsn\") pod \"payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:56.107364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.107345 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lr2m\" (UniqueName: \"kubernetes.io/projected/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-kube-api-access-4lr2m\") pod \"payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:56.107637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.107618 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d707124-abc0-4daa-8a91-0af9e6081951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:56.109209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.109186 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:56.129458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.129429 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d707124-abc0-4daa-8a91-0af9e6081951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:56.153489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.150166 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklsn\" (UniqueName: \"kubernetes.io/projected/1d707124-abc0-4daa-8a91-0af9e6081951-kube-api-access-hklsn\") pod \"payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:56.174482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.167345 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lr2m\" (UniqueName: \"kubernetes.io/projected/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-kube-api-access-4lr2m\") pod \"payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:56.228408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.226898 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:56.240487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:56.232281 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:57.002043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:57.000890 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv"] Oct 31 12:43:57.006835 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:57.006811 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28e1040_0b3d_45c3_aa1b_80a1856a3e10.slice/crio-af93daeaaae5f1120503842e1fdb15437dad621cf3c6c6ba7a6982328ff7ef9f WatchSource:0}: Error finding container af93daeaaae5f1120503842e1fdb15437dad621cf3c6c6ba7a6982328ff7ef9f: Status 404 returned error can't find the container with id af93daeaaae5f1120503842e1fdb15437dad621cf3c6c6ba7a6982328ff7ef9f Oct 31 12:43:57.025904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:57.025802 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:57.291656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:57.290978 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9"] Oct 31 12:43:57.321195 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:57.321163 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" event={"ID":"f28e1040-0b3d-45c3-aa1b-80a1856a3e10","Type":"ContainerStarted","Data":"af93daeaaae5f1120503842e1fdb15437dad621cf3c6c6ba7a6982328ff7ef9f"} Oct 31 12:43:57.323155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:57.323130 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" event={"ID":"1d707124-abc0-4daa-8a91-0af9e6081951","Type":"ContainerStarted","Data":"59135057d2d629e3ae840362588bb56dda061190c88b21586e34a8635c4edc55"} Oct 31 12:43:57.897062 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:57.897020 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:58.333477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.333433 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" event={"ID":"f28e1040-0b3d-45c3-aa1b-80a1856a3e10","Type":"ContainerStarted","Data":"7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a"} Oct 31 12:43:58.333877 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.333533 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:43:58.335614 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.335581 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" event={"ID":"1d707124-abc0-4daa-8a91-0af9e6081951","Type":"ContainerStarted","Data":"bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c"} Oct 31 12:43:58.335765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.335734 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:43:58.354913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.354858 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" podStartSLOduration=3.354847001 podStartE2EDuration="3.354847001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:58.351725753 +0000 UTC m=+949.166740146" watchObservedRunningTime="2026-10-31 12:43:58.354847001 +0000 UTC m=+949.169861392" Oct 31 12:43:58.373402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.373343 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" podStartSLOduration=3.373330407 podStartE2EDuration="3.373330407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:58.369336249 +0000 UTC m=+949.184350665" watchObservedRunningTime="2026-10-31 12:43:58.373330407 +0000 UTC m=+949.188344799" Oct 31 12:43:58.533096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.533037 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" podUID="dfc796eb-38dd-4420-876c-f827d9af160b" containerName="payload-processing" containerID="cri-o://858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081" gracePeriod=25 Oct 31 12:43:58.542934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.542904 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv"] Oct 31 12:43:58.555028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.551819 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.559581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.559444 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv"] Oct 31 12:43:58.567366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.560728 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d702046e-serving-cert\"" Oct 31 12:43:58.620578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.620492 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:58.669539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.667817 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-metrics-certs\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.669539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.667879 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpgl\" (UniqueName: \"kubernetes.io/projected/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-kube-api-access-ffpgl\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.669539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.668056 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.669539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.668137 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-openshift-service-ca\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.768428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.764352 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:58.770198 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.770163 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-metrics-certs\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.770402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.770224 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpgl\" (UniqueName: \"kubernetes.io/projected/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-kube-api-access-ffpgl\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.770402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.770305 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.770402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.770368 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-openshift-service-ca\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.770714 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:58.770618 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-serving-cert: secret "maas-api-e2e-ait-d702046e-serving-cert" not found Oct 31 12:43:58.770714 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:58.770681 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls podName:9567fe0c-63bc-4e0b-b608-e04e3abeec7d nodeName:}" failed. No retries permitted until 2026-10-31 12:43:59.270660652 +0000 UTC m=+950.085675037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls") pod "maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" (UID: "9567fe0c-63bc-4e0b-b608-e04e3abeec7d") : secret "maas-api-e2e-ait-d702046e-serving-cert" not found Oct 31 12:43:58.772692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.771888 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-openshift-service-ca\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.777519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.777496 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-metrics-certs\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.789751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.789321 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpgl\" (UniqueName: \"kubernetes.io/projected/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-kube-api-access-ffpgl\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:58.881776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.881709 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:58.969846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.969809 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:58.973039 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.973013 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc796eb-38dd-4420-876c-f827d9af160b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc796eb-38dd-4420-876c-f827d9af160b-plugins-config-volume\") pod \"dfc796eb-38dd-4420-876c-f827d9af160b\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " Oct 31 12:43:58.973183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.973167 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc796eb-38dd-4420-876c-f827d9af160b-kube-api-access-dtgd9\" (UniqueName: \"kubernetes.io/projected/dfc796eb-38dd-4420-876c-f827d9af160b-kube-api-access-dtgd9\") pod \"dfc796eb-38dd-4420-876c-f827d9af160b\" (UID: \"dfc796eb-38dd-4420-876c-f827d9af160b\") " Oct 31 12:43:58.973404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.973384 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc796eb-38dd-4420-876c-f827d9af160b-plugins-config-volume" pod "dfc796eb-38dd-4420-876c-f827d9af160b" (UID: "dfc796eb-38dd-4420-876c-f827d9af160b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:43:58.974159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.974138 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc796eb-38dd-4420-876c-f827d9af160b-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:58.975626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:58.975604 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc796eb-38dd-4420-876c-f827d9af160b-kube-api-access-dtgd9" pod "dfc796eb-38dd-4420-876c-f827d9af160b" (UID: "dfc796eb-38dd-4420-876c-f827d9af160b"). InnerVolumeSpecName "kube-api-access-dtgd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:43:59.075737 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.075693 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgd9\" (UniqueName: \"kubernetes.io/projected/dfc796eb-38dd-4420-876c-f827d9af160b-kube-api-access-dtgd9\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:43:59.284377 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.284280 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:59.287588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.287564 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls\") pod \"maas-api-e2e-ait-d702046e-7595b9c779-hx5bv\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:59.345188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.345155 6727 generic.go:356] "Generic (PLEG): container finished" podID="dfc796eb-38dd-4420-876c-f827d9af160b" containerID="858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081" exitCode=0 Oct 31 12:43:59.345738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.345208 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" Oct 31 12:43:59.345738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.345239 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" event={"ID":"dfc796eb-38dd-4420-876c-f827d9af160b","Type":"ContainerDied","Data":"858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081"} Oct 31 12:43:59.345738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.345297 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc" event={"ID":"dfc796eb-38dd-4420-876c-f827d9af160b","Type":"ContainerDied","Data":"c9136ee3bd3a18c5c9272dad517260c69eafcb8d482f2ba2d133a7871f3d5dee"} Oct 31 12:43:59.345738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.345327 6727 scope.go:122] "RemoveContainer" containerID="858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081" Oct 31 12:43:59.356748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.356725 6727 scope.go:122] "RemoveContainer" containerID="858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081" Oct 31 12:43:59.357028 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:43:59.357007 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081\": container with ID starting with 858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081 not found: ID does not exist" containerID="858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081" Oct 31 12:43:59.357111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.357041 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081"} err="failed to get container status \"858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081\": rpc error: code = NotFound desc = could not find container \"858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081\": container with ID starting with 858f15aae355b6abf0508c88a4990f26b20e0e19dfe087ce3093063b739cf081 not found: ID does not exist" Oct 31 12:43:59.377234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.377175 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc"] Oct 31 12:43:59.381083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.381058 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-43c7f8da-596f84b497-5drsc"] Oct 31 12:43:59.538386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.538297 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:43:59.574248 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.574205 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:59.855951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.855925 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv"] Oct 31 12:43:59.857530 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:43:59.857455 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9567fe0c_63bc_4e0b_b608_e04e3abeec7d.slice/crio-defc21d3b985152871778be48c0364895a06d090b16c745117b071313d9200e1 WatchSource:0}: Error finding container defc21d3b985152871778be48c0364895a06d090b16c745117b071313d9200e1: Status 404 returned error can't find the container with id defc21d3b985152871778be48c0364895a06d090b16c745117b071313d9200e1 Oct 31 12:43:59.977802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:43:59.977768 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:00.112716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:00.112638 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:00.346880 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:00.346848 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:44:00.355318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:00.355292 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" event={"ID":"9567fe0c-63bc-4e0b-b608-e04e3abeec7d","Type":"ContainerStarted","Data":"defc21d3b985152871778be48c0364895a06d090b16c745117b071313d9200e1"} Oct 31 12:44:00.431675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:00.431647 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:01.114801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:01.114772 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:01.303443 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:01.303409 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc796eb-38dd-4420-876c-f827d9af160b" path="/var/lib/kubelet/pods/dfc796eb-38dd-4420-876c-f827d9af160b/volumes" Oct 31 12:44:01.364980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:01.364899 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" event={"ID":"9567fe0c-63bc-4e0b-b608-e04e3abeec7d","Type":"ContainerStarted","Data":"101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2"} Oct 31 12:44:01.365345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:01.364992 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:44:01.389615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:01.389564 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" podStartSLOduration=2.9445059000000002 podStartE2EDuration="3.389550874s" totalImagesPullingTime="445.044974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:01.384806632 +0000 UTC m=+952.199821023" watchObservedRunningTime="2026-10-31 12:44:01.389550874 +0000 UTC m=+952.204565265" Oct 31 12:44:01.814239 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:01.814154 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:01.986484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:01.980823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:02.132004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:02.131969 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:02.276908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:02.276189 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:02.477260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:02.477148 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:03.033837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.033801 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq"] Oct 31 12:44:03.035017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.034987 6727 cpu_manager.go:414] "containerMap: removing container" podUID="dfc796eb-38dd-4420-876c-f827d9af160b" containerName="payload-processing" Oct 31 12:44:03.035017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.035016 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc796eb-38dd-4420-876c-f827d9af160b" containerName="payload-processing" Oct 31 12:44:03.035319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.035288 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc796eb-38dd-4420-876c-f827d9af160b" containerName="payload-processing" Oct 31 12:44:03.053086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.053061 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.056269 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.056225 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq"] Oct 31 12:44:03.066639 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.066614 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-9b17b9-serving-cert\"" Oct 31 12:44:03.152686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.152631 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:03.196981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.196958 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:44:03.201096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.201069 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.201211 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.201152 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5bvl\" (UniqueName: \"kubernetes.io/projected/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-kube-api-access-l5bvl\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.201368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.201346 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.201426 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.201393 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.299488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.299378 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:03.303220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.303191 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.303370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.303298 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5bvl\" (UniqueName: \"kubernetes.io/projected/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-kube-api-access-l5bvl\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.303612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.303577 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.303739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.303644 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.303947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.303924 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.305970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.305950 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.306066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.306035 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.315386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.315364 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5bvl\" (UniqueName: \"kubernetes.io/projected/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-kube-api-access-l5bvl\") pod \"maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.402969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.394037 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:03.481483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.480948 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:03.932677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:03.932648 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq"] Oct 31 12:44:03.937276 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:44:03.937245 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f24a9a_03ff_49d8_ae6a_b47c7a6791d6.slice/crio-ec31dbeaae01049d2a1d0584c7cdbaed2eadecd150380194f53af0125da08fde WatchSource:0}: Error finding container ec31dbeaae01049d2a1d0584c7cdbaed2eadecd150380194f53af0125da08fde: Status 404 returned error can't find the container with id ec31dbeaae01049d2a1d0584c7cdbaed2eadecd150380194f53af0125da08fde Oct 31 12:44:04.204956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:04.204881 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:44:04.234222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:04.234186 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:04.436486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:04.414576 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:04.559519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:04.559454 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" event={"ID":"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6","Type":"ContainerStarted","Data":"ec31dbeaae01049d2a1d0584c7cdbaed2eadecd150380194f53af0125da08fde"} Oct 31 12:44:04.575264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:04.575050 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:05.353667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:05.353596 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:05.568982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:05.568940 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" event={"ID":"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6","Type":"ContainerStarted","Data":"a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020"} Oct 31 12:44:05.569378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:05.569015 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:05.590359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:05.590307 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" podStartSLOduration=2.198442781 podStartE2EDuration="2.590291901s" totalImagesPullingTime="391.84912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:05.589185038 +0000 UTC m=+956.404199424" watchObservedRunningTime="2026-10-31 12:44:05.590291901 +0000 UTC m=+956.405306291" Oct 31 12:44:06.048876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:06.047835 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:06.714788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:06.714729 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:06.844093 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:06.844066 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:07.386611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:07.386582 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:44:07.479510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:07.478052 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:08.186660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:08.186626 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:08.937533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:08.937447 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:09.348457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:09.348426 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:44:09.348875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:09.348538 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:44:09.824017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:09.823986 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:10.566488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:10.564059 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:11.223718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:11.223682 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:11.584409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:11.584384 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:44:11.910591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:11.910490 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:12.968064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:12.968035 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:13.680054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:13.680014 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:14.229800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:14.229769 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:14.514622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:14.514531 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:14.844614 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:14.844587 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:15.194456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:15.194338 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:15.723640 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:15.723609 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:15.861864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:15.861826 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:16.108535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:16.108500 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:16.520562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:16.520403 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:16.665797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:16.665763 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:17.223334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:17.223295 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:18.053573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:18.041396 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:18.741954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:18.741916 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:44:18.930816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:18.930778 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:19.490879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:19.490844 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:19.650478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:19.650431 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:20.256916 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:20.256628 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:21.917392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:21.917346 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:21.964583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:21.964542 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 31 12:44:22.042586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.014437 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:22.042586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.037650 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.042586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.037695 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.158488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.131048 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.158488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.131716 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:44:22.271390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.261562 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:44:22.271390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.263266 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.271390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.264723 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:44:22.296595 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.293418 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:22.347484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.325841 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:44:22.347484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.327718 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.347484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.339539 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.410826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.398083 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl"] Oct 31 12:44:22.529550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.487797 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:22.608099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.608046 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.772485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.771572 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-2d6201\"" Oct 31 12:44:22.772485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.771733 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-2d6201-dockercfg-rc7kp\"" Oct 31 12:44:22.777535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.773449 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:22.822930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.820142 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:22.822930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.820429 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmntt\" (UniqueName: \"kubernetes.io/projected/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-kube-api-access-cmntt\") pod \"payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:22.870036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.869993 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl"] Oct 31 12:44:22.919070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.919027 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:44:22.939593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.939555 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:44:22.975657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.949363 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:22.976079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.975405 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:22.979312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:22.979278 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmntt\" (UniqueName: \"kubernetes.io/projected/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-kube-api-access-cmntt\") pod \"payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:23.053908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.035131 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmntt\" (UniqueName: \"kubernetes.io/projected/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-kube-api-access-cmntt\") pod \"payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:23.053908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.043618 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:23.053908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.044355 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:44:23.072703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.065601 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:44:23.072703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.065684 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:44:23.137773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.136113 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 31 12:44:23.137773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.136420 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:44:23.138796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.138764 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:44:23.148297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.148267 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:44:23.148613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.148592 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:44:23.191623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.180777 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:44:23.191623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.180838 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:23.227634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.220275 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:23.268708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.268673 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:23.270402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.270371 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:44:23.281564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.268620 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:23.287853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.287828 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:44:23.292497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.290598 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:23.293596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.293561 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:44:23.293831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.293814 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:44:23.344776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.340657 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:23.366503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.340725 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:23.367061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.340757 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:44:23.367897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.328442 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:44:23.753523 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.703014 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:23.921676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:23.915885 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:44:24.042489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.041546 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:44:24.053923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.053872 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:44:24.065765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.065710 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:44:24.217082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.158375 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:24.217082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.177786 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:24.455494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.437768 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:44:24.455494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.452081 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:24.551212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.547335 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xczlv" podUID="3b468147-6eba-4cbd-aff5-d13da0f35497" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:24.708878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.708832 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:24.924634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.924588 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:24.927292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.927260 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:24.992633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.964396 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:44:24.992633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:24.964477 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.049010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.048965 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:44:25.092504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.083568 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.121691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.121626 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.144671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.144612 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 31 12:44:25.460592 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:25.162188 6727 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.818s" Oct 31 12:44:25.565347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.565315 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:44:25.651990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.172318 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:25.651990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.294719 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.651990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.314735 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.651990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.326798 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:44:25.699505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.327795 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:44:25.699505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.460848 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.699505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.461537 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.699505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.461794 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6" podUID="17bb9e9e-099a-4392-87da-ed6a36d161ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.699505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.497142 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr" podUID="bfa9cd02-a108-4237-bc59-eb14f393cd2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.699505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.538201 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:44:25.789762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.789723 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 31 12:44:25.852615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.839526 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:25.935418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.935374 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:25.959842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:25.959798 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:44:26.061543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.049700 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:44:26.127488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.121593 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:44:26.127488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.122828 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:26.198491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.198439 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:44:26.285889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.285694 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:44:26.293711 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.293680 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:44:26.405756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.405721 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:44:26.418652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.408623 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:26.644935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.644892 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:26.654622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.654580 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:44:26.655109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.654738 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:44:26.737904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.737135 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:44:26.738815 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.738481 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:44:26.740808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.738509 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:44:26.747670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.747641 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:44:26.963544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.946544 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:26.963544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:26.961149 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:27.045804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.014882 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" event={"ID":"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21","Type":"ContainerStarted","Data":"96b6ea8b2c170fc6957077ba763d26ed4d2eeb7b7ed17433f4d8c2b679019852"} Oct 31 12:44:27.045804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.017484 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:27.064678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.064635 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:27.142627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.124570 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:44:27.142627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.124917 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:44:27.587557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.564015 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m"] Oct 31 12:44:27.614632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.614582 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx"] Oct 31 12:44:27.645532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.645489 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:27.659371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.659328 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:27.659882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.659851 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:44:27.775060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.742296 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:27.775060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.748533 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:44:27.775060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.749680 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:44:27.775060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.752899 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:27.862612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.861296 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl"] Oct 31 12:44:27.977179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.976996 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 12:44:27.981188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:27.981157 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:44:28.010122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.008997 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:44:28.011869 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.010650 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:44:28.052620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.050994 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:44:28.066225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.066195 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:28.155123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.066642 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:44:28.155482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.155452 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:44:28.156818 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.151532 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:44:28.221616 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.221573 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:28.318709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.236692 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:44:28.319276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.319251 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:44:28.319494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.319481 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:44:28.319648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.319638 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:44:28.358834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.358449 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:44:28.373772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.373748 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:44:28.378200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.378177 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:44:28.622506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:28.615105 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:29.157194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:29.157159 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" event={"ID":"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21","Type":"ContainerStarted","Data":"4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92"} Oct 31 12:44:29.157660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:29.157316 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:29.195678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:29.183441 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" podStartSLOduration=7.183423108 podStartE2EDuration="7.183423108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:29.175865067 +0000 UTC m=+979.990879460" watchObservedRunningTime="2026-10-31 12:44:29.183423108 +0000 UTC m=+979.998437500" Oct 31 12:44:29.325033 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:29.324996 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:30.200076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:30.199958 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:30.354779 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:30.354746 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:44:31.077155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:31.077113 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:31.807153 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:31.807119 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:32.284178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.284145 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv"] Oct 31 12:44:32.284656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.284622 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" podUID="9567fe0c-63bc-4e0b-b608-e04e3abeec7d" containerName="maas-api" containerID="cri-o://101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2" gracePeriod=30 Oct 31 12:44:32.364889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.364853 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:32.380787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.380723 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" podUID="9567fe0c-63bc-4e0b-b608-e04e3abeec7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.93:8443/health\": dial tcp 10.134.0.93:8443: connect: connection refused" Oct 31 12:44:32.541694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.540082 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:32.629376 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.625328 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" podUID="14fd6fca-fc28-4180-8ef5-18454fb4265b" containerName="payload-pre-processing" containerID="cri-o://32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16" gracePeriod=25 Oct 31 12:44:32.629376 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.625549 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" podUID="83258142-61b9-4760-a622-59ea097fd738" containerName="payload-processing" containerID="cri-o://199a1c06cdab33e7ef9f71198561309d3622429b523b1c3510515f663a566785" gracePeriod=25 Oct 31 12:44:32.699485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.696551 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:32.837273 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.837174 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:32.938514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:32.936017 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:44:33.022858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.022822 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-metrics-certs\") pod \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " Oct 31 12:44:33.027406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.027368 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-openshift-service-ca\") pod \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " Oct 31 12:44:33.027696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.027675 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls\") pod \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " Oct 31 12:44:33.030157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.028096 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-kube-api-access-ffpgl\" (UniqueName: \"kubernetes.io/projected/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-kube-api-access-ffpgl\") pod \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\" (UID: \"9567fe0c-63bc-4e0b-b608-e04e3abeec7d\") " Oct 31 12:44:33.046119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.043746 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-kube-api-access-ffpgl" pod "9567fe0c-63bc-4e0b-b608-e04e3abeec7d" (UID: "9567fe0c-63bc-4e0b-b608-e04e3abeec7d"). InnerVolumeSpecName "kube-api-access-ffpgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:44:33.048302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.048185 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-metrics-certs" pod "9567fe0c-63bc-4e0b-b608-e04e3abeec7d" (UID: "9567fe0c-63bc-4e0b-b608-e04e3abeec7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:44:33.082304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.080960 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffpgl\" (UniqueName: \"kubernetes.io/projected/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-kube-api-access-ffpgl\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.082304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.080995 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.089756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.088580 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-openshift-service-ca" pod "9567fe0c-63bc-4e0b-b608-e04e3abeec7d" (UID: "9567fe0c-63bc-4e0b-b608-e04e3abeec7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:44:33.111860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.110647 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls" pod "9567fe0c-63bc-4e0b-b608-e04e3abeec7d" (UID: "9567fe0c-63bc-4e0b-b608-e04e3abeec7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:44:33.168707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.158578 6727 patch_prober.go:29] interesting pod/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:44:33.168707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.158684 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" podUID="1906916a-b650-415c-b664-80303d614c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:44:33.183706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.183668 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.183706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.183709 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9567fe0c-63bc-4e0b-b608-e04e3abeec7d-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.186208 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.186179 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:44:33.222680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.222643 6727 generic.go:356] "Generic (PLEG): container finished" podID="9567fe0c-63bc-4e0b-b608-e04e3abeec7d" containerID="101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2" exitCode=0 Oct 31 12:44:33.223220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.223194 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" event={"ID":"9567fe0c-63bc-4e0b-b608-e04e3abeec7d","Type":"ContainerDied","Data":"101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2"} Oct 31 12:44:33.223412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.223393 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" event={"ID":"9567fe0c-63bc-4e0b-b608-e04e3abeec7d","Type":"ContainerDied","Data":"defc21d3b985152871778be48c0364895a06d090b16c745117b071313d9200e1"} Oct 31 12:44:33.223562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.223547 6727 scope.go:122] "RemoveContainer" containerID="101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2" Oct 31 12:44:33.224226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.224205 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv" Oct 31 12:44:33.232554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.232429 6727 generic.go:356] "Generic (PLEG): container finished" podID="14fd6fca-fc28-4180-8ef5-18454fb4265b" containerID="32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16" exitCode=0 Oct 31 12:44:33.233399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.232843 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" event={"ID":"14fd6fca-fc28-4180-8ef5-18454fb4265b","Type":"ContainerDied","Data":"32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16"} Oct 31 12:44:33.233399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.232894 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" event={"ID":"14fd6fca-fc28-4180-8ef5-18454fb4265b","Type":"ContainerDied","Data":"f669be9a29ea80b35cdcb6cd32afdc35f602f6ac2fc91c77a41c571ee666d0f4"} Oct 31 12:44:33.233399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.232966 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx" Oct 31 12:44:33.234452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.234159 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw"] Oct 31 12:44:33.235780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.235665 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" podUID="1906916a-b650-415c-b664-80303d614c70" containerName="istio-proxy" containerID="cri-o://85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e" gracePeriod=30 Oct 31 12:44:33.237403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.237218 6727 generic.go:356] "Generic (PLEG): container finished" podID="83258142-61b9-4760-a622-59ea097fd738" containerID="199a1c06cdab33e7ef9f71198561309d3622429b523b1c3510515f663a566785" exitCode=0 Oct 31 12:44:33.237403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.237356 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" event={"ID":"83258142-61b9-4760-a622-59ea097fd738","Type":"ContainerDied","Data":"199a1c06cdab33e7ef9f71198561309d3622429b523b1c3510515f663a566785"} Oct 31 12:44:33.248868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.248844 6727 scope.go:122] "RemoveContainer" containerID="101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2" Oct 31 12:44:33.249239 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:33.249210 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2\": container with ID starting with 101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2 not found: ID does not exist" containerID="101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2" Oct 31 12:44:33.249353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.249252 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2"} err="failed to get container status \"101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2\": rpc error: code = NotFound desc = could not find container \"101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2\": container with ID starting with 101cd877f7ab78db39a4d2268509a719f476892ab37b0f0ae725efade9a13ec2 not found: ID does not exist" Oct 31 12:44:33.249353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.249275 6727 scope.go:122] "RemoveContainer" containerID="32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16" Oct 31 12:44:33.272488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.272163 6727 scope.go:122] "RemoveContainer" containerID="32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16" Oct 31 12:44:33.272861 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:33.272765 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16\": container with ID starting with 32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16 not found: ID does not exist" containerID="32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16" Oct 31 12:44:33.272861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.272802 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16"} err="failed to get container status \"32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16\": rpc error: code = NotFound desc = could not find container \"32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16\": container with ID starting with 32efc5064814673d0f6623b7a84b28af42dabff39db9ca92afbfa91caca7cb16 not found: ID does not exist" Oct 31 12:44:33.281811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.281784 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv"] Oct 31 12:44:33.285424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.285398 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14fd6fca-fc28-4180-8ef5-18454fb4265b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fd6fca-fc28-4180-8ef5-18454fb4265b-plugins-config-volume\") pod \"14fd6fca-fc28-4180-8ef5-18454fb4265b\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " Oct 31 12:44:33.285845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.285826 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fd6fca-fc28-4180-8ef5-18454fb4265b-kube-api-access-mzmhx\" (UniqueName: \"kubernetes.io/projected/14fd6fca-fc28-4180-8ef5-18454fb4265b-kube-api-access-mzmhx\") pod \"14fd6fca-fc28-4180-8ef5-18454fb4265b\" (UID: \"14fd6fca-fc28-4180-8ef5-18454fb4265b\") " Oct 31 12:44:33.288769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.288726 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fd6fca-fc28-4180-8ef5-18454fb4265b-plugins-config-volume" pod "14fd6fca-fc28-4180-8ef5-18454fb4265b" (UID: "14fd6fca-fc28-4180-8ef5-18454fb4265b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:44:33.289256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.289232 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d702046e-7595b9c779-hx5bv"] Oct 31 12:44:33.304335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.304287 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:33.313561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.312652 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9567fe0c-63bc-4e0b-b608-e04e3abeec7d" path="/var/lib/kubelet/pods/9567fe0c-63bc-4e0b-b608-e04e3abeec7d/volumes" Oct 31 12:44:33.313561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.312731 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fd6fca-fc28-4180-8ef5-18454fb4265b-kube-api-access-mzmhx" pod "14fd6fca-fc28-4180-8ef5-18454fb4265b" (UID: "14fd6fca-fc28-4180-8ef5-18454fb4265b"). InnerVolumeSpecName "kube-api-access-mzmhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:44:33.431533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.431503 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzmhx\" (UniqueName: \"kubernetes.io/projected/14fd6fca-fc28-4180-8ef5-18454fb4265b-kube-api-access-mzmhx\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.439487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.431988 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fd6fca-fc28-4180-8ef5-18454fb4265b-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.575863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.575753 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:44:33.594833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.593704 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj"] Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595138 6727 cpu_manager.go:414] "containerMap: removing container" podUID="83258142-61b9-4760-a622-59ea097fd738" containerName="payload-processing" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595164 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83258142-61b9-4760-a622-59ea097fd738" containerName="payload-processing" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595231 6727 cpu_manager.go:414] "containerMap: removing container" podUID="14fd6fca-fc28-4180-8ef5-18454fb4265b" containerName="payload-pre-processing" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595239 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fd6fca-fc28-4180-8ef5-18454fb4265b" containerName="payload-pre-processing" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595332 6727 cpu_manager.go:414] "containerMap: removing container" podUID="9567fe0c-63bc-4e0b-b608-e04e3abeec7d" containerName="maas-api" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595342 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9567fe0c-63bc-4e0b-b608-e04e3abeec7d" containerName="maas-api" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595555 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fd6fca-fc28-4180-8ef5-18454fb4265b" containerName="payload-pre-processing" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595576 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9567fe0c-63bc-4e0b-b608-e04e3abeec7d" containerName="maas-api" Oct 31 12:44:33.596311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.595592 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="83258142-61b9-4760-a622-59ea097fd738" containerName="payload-processing" Oct 31 12:44:33.608355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.601882 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj"] Oct 31 12:44:33.608355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.601972 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.608355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.604569 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j"] Oct 31 12:44:33.616658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.616632 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.648432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.647550 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx"] Oct 31 12:44:33.651540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.651515 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83258142-61b9-4760-a622-59ea097fd738-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83258142-61b9-4760-a622-59ea097fd738-plugins-config-volume\") pod \"83258142-61b9-4760-a622-59ea097fd738\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " Oct 31 12:44:33.651971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.651953 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83258142-61b9-4760-a622-59ea097fd738-kube-api-access-rg4nl\" (UniqueName: \"kubernetes.io/projected/83258142-61b9-4760-a622-59ea097fd738-kube-api-access-rg4nl\") pod \"83258142-61b9-4760-a622-59ea097fd738\" (UID: \"83258142-61b9-4760-a622-59ea097fd738\") " Oct 31 12:44:33.652552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.652406 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83258142-61b9-4760-a622-59ea097fd738-plugins-config-volume" pod "83258142-61b9-4760-a622-59ea097fd738" (UID: "83258142-61b9-4760-a622-59ea097fd738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:44:33.652753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.652727 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.652842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.652773 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.652911 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.653070 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.653120 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.653137 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xg8\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-kube-api-access-64xg8\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.653190 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.653248 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.653296 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.653458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.653391 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83258142-61b9-4760-a622-59ea097fd738-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.659281 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.659250 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83258142-61b9-4760-a622-59ea097fd738-kube-api-access-rg4nl" pod "83258142-61b9-4760-a622-59ea097fd738" (UID: "83258142-61b9-4760-a622-59ea097fd738"). InnerVolumeSpecName "kube-api-access-rg4nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:44:33.686859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.686827 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j"] Oct 31 12:44:33.688122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.688094 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d702046e-856bf876f5-hpcbx"] Oct 31 12:44:33.695618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.695554 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ed25a690-openshift-default-dockercfg-wwhr7\"" Oct 31 12:44:33.755304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.755270 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.755534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.755347 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.755534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.755382 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b282d2-95dc-4f43-84c6-77271c02e92e-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.755534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.755444 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.755534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.755483 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.755868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.755842 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.755950 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.755858 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756019 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756107 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756071 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756113 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.756226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756200 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756284 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756288 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shbb8\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-kube-api-access-shbb8\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.756544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756530 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756301 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756586 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.756678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756625 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756650 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xg8\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-kube-api-access-64xg8\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756684 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.756776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756744 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.756878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756816 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.756934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756877 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.756934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.756927 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.757296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.757081 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4nl\" (UniqueName: \"kubernetes.io/projected/83258142-61b9-4760-a622-59ea097fd738-kube-api-access-rg4nl\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:33.759454 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.759431 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.760176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.760149 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.778826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.778795 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.779178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.779153 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xg8\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-kube-api-access-64xg8\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.858810 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.858775 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.858894 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859032 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shbb8\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-kube-api-access-shbb8\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859277 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859315 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859383 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859414 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859451 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859528 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859557 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859740 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859844 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.859995 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b282d2-95dc-4f43-84c6-77271c02e92e-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.860761 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b282d2-95dc-4f43-84c6-77271c02e92e-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.863807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.862945 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.863807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.863001 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.866280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.866249 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq"] Oct 31 12:44:33.872533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.872506 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:33.885402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.882447 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.885402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.882489 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-2d6201-serving-cert\"" Oct 31 12:44:33.885402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.882905 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shbb8\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-kube-api-access-shbb8\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:33.897401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.897320 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq"] Oct 31 12:44:33.953209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.953144 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:33.961054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.961020 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf7zc\" (UniqueName: \"kubernetes.io/projected/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-kube-api-access-bf7zc\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:33.961203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.961070 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:33.961373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.961357 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:33.961434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.961395 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:33.968909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.968885 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:33.971750 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:33.971730 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:34.068132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.068097 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf7zc\" (UniqueName: \"kubernetes.io/projected/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-kube-api-access-bf7zc\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.068345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.068155 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.068502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.068446 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.068607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.068505 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.070548 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:34.068780 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-serving-cert: secret "maas-api-e2e-shared-b-w6-2d6201-serving-cert" not found Oct 31 12:44:34.070548 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:34.068863 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls podName:7543a97c-b2c1-46ef-82bc-57bb1ecd2e67 nodeName:}" failed. No retries permitted until 2026-10-31 12:44:34.568836753 +0000 UTC m=+985.383851122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls") pod "maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" (UID: "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67") : secret "maas-api-e2e-shared-b-w6-2d6201-serving-cert" not found Oct 31 12:44:34.070548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.070290 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.075190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.074968 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.087030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.087002 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf7zc\" (UniqueName: \"kubernetes.io/projected/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-kube-api-access-bf7zc\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.108079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.108043 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:34.261830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.261691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:34.267776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.267751 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" Oct 31 12:44:34.267918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.267751 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m" event={"ID":"83258142-61b9-4760-a622-59ea097fd738","Type":"ContainerDied","Data":"064e7edce5cc4a48a43592ec051fdc8de4eae3e13d063d951da80798bc741973"} Oct 31 12:44:34.267960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.267919 6727 scope.go:122] "RemoveContainer" containerID="199a1c06cdab33e7ef9f71198561309d3622429b523b1c3510515f663a566785" Oct 31 12:44:34.299027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.298980 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m"] Oct 31 12:44:34.303636 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.303607 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d702046e-7f9c6b774c-8dl9m"] Oct 31 12:44:34.359139 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:44:34.359112 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6ad2e_85f2_4148_a9a8_d7c53d72d286.slice/crio-b4eb4adfe7bce864c1e9b5cce1db3391fa187f431ac27a2739a2421e23da41e0 WatchSource:0}: Error finding container b4eb4adfe7bce864c1e9b5cce1db3391fa187f431ac27a2739a2421e23da41e0: Status 404 returned error can't find the container with id b4eb4adfe7bce864c1e9b5cce1db3391fa187f431ac27a2739a2421e23da41e0 Oct 31 12:44:34.359328 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.359307 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj"] Oct 31 12:44:34.361905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.361872 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:44:34.361998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.361933 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:44:34.361998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.361961 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:44:34.378320 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:44:34.378293 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b282d2_95dc_4f43_84c6_77271c02e92e.slice/crio-ccb4555464b1e37911f97b1476a7ff3122f093037de296b312241194cf01396a WatchSource:0}: Error finding container ccb4555464b1e37911f97b1476a7ff3122f093037de296b312241194cf01396a: Status 404 returned error can't find the container with id ccb4555464b1e37911f97b1476a7ff3122f093037de296b312241194cf01396a Oct 31 12:44:34.382007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.381982 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j"] Oct 31 12:44:34.382357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.382325 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:44:34.382415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.382386 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:44:34.382448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.382417 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:44:34.552869 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.552766 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:34.582679 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.582203 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.620127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.619273 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:34.804533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:34.803889 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:35.126326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.126238 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:35.289912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.289876 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" event={"ID":"d7b282d2-95dc-4f43-84c6-77271c02e92e","Type":"ContainerStarted","Data":"fd8a5f296ae2279dc1d02ab83a5fa9a96c9a7e1cf80c0e73506efa5be3485916"} Oct 31 12:44:35.290179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.290153 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" event={"ID":"d7b282d2-95dc-4f43-84c6-77271c02e92e","Type":"ContainerStarted","Data":"ccb4555464b1e37911f97b1476a7ff3122f093037de296b312241194cf01396a"} Oct 31 12:44:35.291738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.291710 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:35.307285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.307257 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fd6fca-fc28-4180-8ef5-18454fb4265b" path="/var/lib/kubelet/pods/14fd6fca-fc28-4180-8ef5-18454fb4265b/volumes" Oct 31 12:44:35.307854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.307831 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83258142-61b9-4760-a622-59ea097fd738" path="/var/lib/kubelet/pods/83258142-61b9-4760-a622-59ea097fd738/volumes" Oct 31 12:44:35.308225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.308202 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" event={"ID":"79b6ad2e-85f2-4148-a9a8-d7c53d72d286","Type":"ContainerStarted","Data":"f2460625c80c469fc7dc05c562aaa4a8f6d7df5b55a463ba2a9f8d8d820980dd"} Oct 31 12:44:35.308312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.308261 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" event={"ID":"79b6ad2e-85f2-4148-a9a8-d7c53d72d286","Type":"ContainerStarted","Data":"b4eb4adfe7bce864c1e9b5cce1db3391fa187f431ac27a2739a2421e23da41e0"} Oct 31 12:44:35.319825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.319773 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podStartSLOduration=2.319757455 podStartE2EDuration="2.319757455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:35.3171733 +0000 UTC m=+986.132187703" watchObservedRunningTime="2026-10-31 12:44:35.319757455 +0000 UTC m=+986.134771845" Oct 31 12:44:35.344389 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.344321 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podStartSLOduration=2.34430114 podStartE2EDuration="2.34430114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:35.340522672 +0000 UTC m=+986.155537059" watchObservedRunningTime="2026-10-31 12:44:35.34430114 +0000 UTC m=+986.159315531" Oct 31 12:44:35.424109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.424045 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq"] Oct 31 12:44:35.427453 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:44:35.427414 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7543a97c_b2c1_46ef_82bc_57bb1ecd2e67.slice/crio-7382723185b31f1504d9330488614fa1266b839407a7ec2e791ce223bc908634 WatchSource:0}: Error finding container 7382723185b31f1504d9330488614fa1266b839407a7ec2e791ce223bc908634: Status 404 returned error can't find the container with id 7382723185b31f1504d9330488614fa1266b839407a7ec2e791ce223bc908634 Oct 31 12:44:35.990187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.969781 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:35.990187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.980128 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:35.990756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.990619 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 12:44:35.990756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.990678 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 12:44:36.004512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.991621 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 12:44:36.004512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:35.991667 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 12:44:36.327944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:36.327300 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" event={"ID":"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67","Type":"ContainerStarted","Data":"0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc"} Oct 31 12:44:36.327944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:36.327394 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:36.327944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:36.327440 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" event={"ID":"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67","Type":"ContainerStarted","Data":"7382723185b31f1504d9330488614fa1266b839407a7ec2e791ce223bc908634"} Oct 31 12:44:36.350661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:36.349332 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" podStartSLOduration=2.850780255 podStartE2EDuration="3.349318292s" totalImagesPullingTime="498.538037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:36.346919832 +0000 UTC m=+987.161934224" watchObservedRunningTime="2026-10-31 12:44:36.349318292 +0000 UTC m=+987.164332683" Oct 31 12:44:36.375494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:36.365027 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:37.002519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:37.000559 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 12:44:37.002519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:37.000635 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 12:44:37.002519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:37.001017 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 12:44:37.002519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:37.001071 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 12:44:37.327159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:37.327127 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:38.010544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:37.999265 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 12:44:38.010544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:37.999319 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 12:44:38.033523 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.020742 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 12:44:38.033523 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.020830 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 12:44:38.041563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.038850 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.085710 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1906916a-b650-415c-b664-80303d614c70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1906916a-b650-415c-b664-80303d614c70-istiod-ca-cert\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.085804 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-socket\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.085916 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-certs\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.085966 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-data\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.086177 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-credential-socket\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.086273 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-envoy\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.086326 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-kube-api-access-975xv\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-kube-api-access-975xv\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.086373 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1906916a-b650-415c-b664-80303d614c70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1906916a-b650-415c-b664-80303d614c70-istio-podinfo\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.086484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.086496 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-istio-token\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-istio-token\") pod \"1906916a-b650-415c-b664-80303d614c70\" (UID: \"1906916a-b650-415c-b664-80303d614c70\") " Oct 31 12:44:38.128064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.124324 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-data" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:44:38.128064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.124745 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1906916a-b650-415c-b664-80303d614c70-istiod-ca-cert" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:44:38.128064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.124985 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-socket" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:44:38.128064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.125159 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-certs" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:44:38.128064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.126600 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-credential-socket" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:44:38.137819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.137762 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-istio-token" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:44:38.138286 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.138096 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-kube-api-access-975xv" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "kube-api-access-975xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:44:38.147399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.147328 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1906916a-b650-415c-b664-80303d614c70-istio-podinfo" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:44:38.152593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.151665 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-envoy" pod "1906916a-b650-415c-b664-80303d614c70" (UID: "1906916a-b650-415c-b664-80303d614c70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192834 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192870 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1906916a-b650-415c-b664-80303d614c70-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192887 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192903 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192919 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192933 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192947 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1906916a-b650-415c-b664-80303d614c70-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192962 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-975xv\" (UniqueName: \"kubernetes.io/projected/1906916a-b650-415c-b664-80303d614c70-kube-api-access-975xv\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.195539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.192978 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1906916a-b650-415c-b664-80303d614c70-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:44:38.294488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.293671 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:38.348905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.348872 6727 generic.go:356] "Generic (PLEG): container finished" podID="1906916a-b650-415c-b664-80303d614c70" containerID="85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e" exitCode=0 Oct 31 12:44:38.349058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.348987 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" event={"ID":"1906916a-b650-415c-b664-80303d614c70","Type":"ContainerDied","Data":"85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e"} Oct 31 12:44:38.349058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.349026 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" event={"ID":"1906916a-b650-415c-b664-80303d614c70","Type":"ContainerDied","Data":"5a4ddf39986ada3b2cf444b7d29d19ad61ecb1d49b79375548ed2e78039230ae"} Oct 31 12:44:38.349058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.349046 6727 scope.go:122] "RemoveContainer" containerID="85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e" Oct 31 12:44:38.349180 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.349153 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw" Oct 31 12:44:38.361497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.361448 6727 scope.go:122] "RemoveContainer" containerID="85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e" Oct 31 12:44:38.361779 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:38.361759 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e\": container with ID starting with 85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e not found: ID does not exist" containerID="85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e" Oct 31 12:44:38.361852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.361790 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e"} err="failed to get container status \"85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e\": rpc error: code = NotFound desc = could not find container \"85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e\": container with ID starting with 85fe040c31b3c327de1fe73e47c1037e233e816d14d1469e37dca440c7b5cc7e not found: ID does not exist" Oct 31 12:44:38.443003 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.442966 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:38.473681 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.472115 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw"] Oct 31 12:44:38.504577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:38.503249 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d702046e-openshift-default-fb99775bf-wljxw"] Oct 31 12:44:39.003502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.003032 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 12:44:39.003502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.003092 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 12:44:39.025872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.025723 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 12:44:39.025872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.025779 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 12:44:39.335349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.335300 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1906916a-b650-415c-b664-80303d614c70" path="/var/lib/kubelet/pods/1906916a-b650-415c-b664-80303d614c70/volumes" Oct 31 12:44:39.553327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.553293 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:39.634555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.634118 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk"] Oct 31 12:44:39.636926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.635934 6727 cpu_manager.go:414] "containerMap: removing container" podUID="1906916a-b650-415c-b664-80303d614c70" containerName="istio-proxy" Oct 31 12:44:39.636926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.635965 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1906916a-b650-415c-b664-80303d614c70" containerName="istio-proxy" Oct 31 12:44:39.636926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.636127 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1906916a-b650-415c-b664-80303d614c70" containerName="istio-proxy" Oct 31 12:44:39.642282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.642257 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk"] Oct 31 12:44:39.642486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.642374 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:39.647139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.647110 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd"] Oct 31 12:44:39.653708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.653387 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:39.672489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.660528 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd"] Oct 31 12:44:39.672489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.661750 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-62734c\"" Oct 31 12:44:39.683504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.674881 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-62734c-dockercfg-4q5st\"" Oct 31 12:44:39.808518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.806364 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:39.879298 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.879247 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:39.879537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.879515 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrl8\" (UniqueName: \"kubernetes.io/projected/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-kube-api-access-shrl8\") pod \"payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:39.973571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.973131 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 12:44:39.973571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.973195 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 12:44:40.082586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.979559 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 12:44:40.082586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:39.979629 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 12:44:40.133502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.084482 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf7b01-8d02-47e5-877f-addd475b2921-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:40.133502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.084605 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:40.133502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.084635 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbgz\" (UniqueName: \"kubernetes.io/projected/badf7b01-8d02-47e5-877f-addd475b2921-kube-api-access-flbgz\") pod \"payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:40.133502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.084841 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrl8\" (UniqueName: \"kubernetes.io/projected/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-kube-api-access-shrl8\") pod \"payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:40.133502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.086234 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:40.187786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.187601 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf7b01-8d02-47e5-877f-addd475b2921-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:40.187786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.187716 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbgz\" (UniqueName: \"kubernetes.io/projected/badf7b01-8d02-47e5-877f-addd475b2921-kube-api-access-flbgz\") pod \"payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:40.222856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.222784 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 12:44:40.226039 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.225950 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrl8\" (UniqueName: \"kubernetes.io/projected/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-kube-api-access-shrl8\") pod \"payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:40.238390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.238339 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf7b01-8d02-47e5-877f-addd475b2921-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:40.238390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.238388 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbgz\" (UniqueName: \"kubernetes.io/projected/badf7b01-8d02-47e5-877f-addd475b2921-kube-api-access-flbgz\") pod \"payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:40.325066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.324933 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:40.431501 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.414911 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:40.530351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.530261 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:40.922701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.922532 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd"] Oct 31 12:44:40.992879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.984689 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 12:44:40.992879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.984760 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 12:44:40.992879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.987043 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 12:44:40.992879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:40.987085 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 12:44:41.148452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.147892 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk"] Oct 31 12:44:41.311005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.310971 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:41.547542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.540092 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" event={"ID":"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e","Type":"ContainerStarted","Data":"fb2bb934a78206ea1ef26f639c24c75efd434cfe3cad51875ac35ac3dc47bde9"} Oct 31 12:44:41.547542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.540163 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" event={"ID":"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e","Type":"ContainerStarted","Data":"15d703bf04dceb30c009366ff42881152b904e4b18bdd0fd4be1bfb3050c881b"} Oct 31 12:44:41.547542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.541382 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:41.575662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.575599 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" podStartSLOduration=2.575576151 podStartE2EDuration="2.575576151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:41.563699402 +0000 UTC m=+992.378713794" watchObservedRunningTime="2026-10-31 12:44:41.575576151 +0000 UTC m=+992.390590543" Oct 31 12:44:41.591173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.590976 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" event={"ID":"badf7b01-8d02-47e5-877f-addd475b2921","Type":"ContainerStarted","Data":"832074edc2bcdd78b1d7e611e03b440ccc3f7496edfac700ae79cb4c5d242b01"} Oct 31 12:44:41.591173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.591047 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:41.591173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.591073 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" event={"ID":"badf7b01-8d02-47e5-877f-addd475b2921","Type":"ContainerStarted","Data":"545bdd0181141dd314d20cd373e292d96cb7fbaa847cba1d37fe3a8bd46b7163"} Oct 31 12:44:41.969766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.969731 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 12:44:41.969953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:41.969785 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 12:44:42.072481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.072193 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:42.075478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.073844 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:42.088236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.088198 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:44:42.116574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.116529 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" podStartSLOduration=3.116514232 podStartE2EDuration="3.116514232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:41.64068537 +0000 UTC m=+992.455699763" watchObservedRunningTime="2026-10-31 12:44:42.116514232 +0000 UTC m=+992.931528614" Oct 31 12:44:42.310795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.310704 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:42.364852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.364821 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 12:44:42.579731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.579691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:42.703520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.703478 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:42.980542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.979941 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:42.991919 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.989076 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:43.000485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:42.993395 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:44:43.355714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:43.355641 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:44.017899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:44.017865 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:44.748767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:44.748739 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:45.415284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:45.415252 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:46.506047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:46.506003 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:47.336746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:47.336701 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:47.968535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:47.968500 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w"] Oct 31 12:44:47.973871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:47.973846 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:47.991933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:47.991904 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w"] Oct 31 12:44:47.992129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:47.992107 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-62734c-serving-cert\"" Oct 31 12:44:48.058676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.058624 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:48.081331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.076068 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqjd\" (UniqueName: \"kubernetes.io/projected/fda97a76-3a20-4ee9-9a41-dfdc3887df90-kube-api-access-vxqjd\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.081868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.081833 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.082189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.082167 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.082400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.082380 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda97a76-3a20-4ee9-9a41-dfdc3887df90-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.184362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.184319 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqjd\" (UniqueName: \"kubernetes.io/projected/fda97a76-3a20-4ee9-9a41-dfdc3887df90-kube-api-access-vxqjd\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.184625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.184479 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.184625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.184546 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.184625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.184581 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda97a76-3a20-4ee9-9a41-dfdc3887df90-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.184841 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:48.184666 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-serving-cert: secret "maas-api-e2e-shared-a-w4-62734c-serving-cert" not found Oct 31 12:44:48.184841 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:48.184737 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls podName:fda97a76-3a20-4ee9-9a41-dfdc3887df90 nodeName:}" failed. No retries permitted until 2026-10-31 12:44:48.684716282 +0000 UTC m=+999.499730669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls") pod "maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" (UID: "fda97a76-3a20-4ee9-9a41-dfdc3887df90") : secret "maas-api-e2e-shared-a-w4-62734c-serving-cert" not found Oct 31 12:44:48.185459 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.185433 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda97a76-3a20-4ee9-9a41-dfdc3887df90-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.187857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.187829 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.197695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.197656 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqjd\" (UniqueName: \"kubernetes.io/projected/fda97a76-3a20-4ee9-9a41-dfdc3887df90-kube-api-access-vxqjd\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.201751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.201724 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:48.695344 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.695309 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:48.696066 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:48.695681 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-serving-cert: secret "maas-api-e2e-shared-a-w4-62734c-serving-cert" not found Oct 31 12:44:48.696271 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:44:48.696259 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls podName:fda97a76-3a20-4ee9-9a41-dfdc3887df90 nodeName:}" failed. No retries permitted until 2026-10-31 12:44:49.696236373 +0000 UTC m=+1000.511250761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls") pod "maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" (UID: "fda97a76-3a20-4ee9-9a41-dfdc3887df90") : secret "maas-api-e2e-shared-a-w4-62734c-serving-cert" not found Oct 31 12:44:48.869797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:48.869765 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:49.018813 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:49.018686 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:49.714388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:49.714299 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:49.716881 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:49.716859 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:49.749357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:49.749320 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:49.797148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:49.795608 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:49.933941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:49.933907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:50.113563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:50.113536 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w"] Oct 31 12:44:50.113999 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:44:50.113971 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda97a76_3a20_4ee9_9a41_dfdc3887df90.slice/crio-cc2375a2dba19d426eae8e9c3f48c28caca1494af2e1d018257773a8a293ecc3 WatchSource:0}: Error finding container cc2375a2dba19d426eae8e9c3f48c28caca1494af2e1d018257773a8a293ecc3: Status 404 returned error can't find the container with id cc2375a2dba19d426eae8e9c3f48c28caca1494af2e1d018257773a8a293ecc3 Oct 31 12:44:50.469813 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:50.469685 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:50.709557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:50.709495 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" event={"ID":"fda97a76-3a20-4ee9-9a41-dfdc3887df90","Type":"ContainerStarted","Data":"df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97"} Oct 31 12:44:50.709754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:50.709564 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" event={"ID":"fda97a76-3a20-4ee9-9a41-dfdc3887df90","Type":"ContainerStarted","Data":"cc2375a2dba19d426eae8e9c3f48c28caca1494af2e1d018257773a8a293ecc3"} Oct 31 12:44:50.709754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:50.709723 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:50.733521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:50.733395 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" podStartSLOduration=3.332208252 podStartE2EDuration="3.73338109s" totalImagesPullingTime="401.172838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:50.729406676 +0000 UTC m=+1001.544421067" watchObservedRunningTime="2026-10-31 12:44:50.73338109 +0000 UTC m=+1001.548395480" Oct 31 12:44:51.223546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:51.223502 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:52.021238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:52.021202 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:52.600085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:52.600058 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:44:52.673858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:52.673825 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:53.356658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:53.356583 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:53.486952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:53.486917 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:53.607940 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:53.607863 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:44:54.047696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:54.047625 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:54.734083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:54.734048 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:55.514670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:55.514640 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:56.236044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:56.235277 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:56.723853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:56.723820 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:44:57.052082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:57.051895 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:58.084092 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:58.084048 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:58.237669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:58.237640 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:58.679285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:58.679254 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:58.909358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:58.909328 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:59.139604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:59.139571 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:59.743254 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:59.743213 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:59.885969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:44:59.885930 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:00.350171 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:45:00.350137 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:45:00.573148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:00.573052 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:01.199706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:01.199674 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:01.341524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:01.341445 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:01.688579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:01.688542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:02.300777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:02.300731 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:02.521063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:02.521013 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:03.030445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:03.030398 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:03.171930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:03.171899 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:03.446772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:03.446738 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:04.256449 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.256419 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:04.295035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.295004 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll"] Oct 31 12:45:04.300054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.300033 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.313963 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.313937 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-8faad6-dockercfg-twcjd\"" Oct 31 12:45:04.316485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.316438 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll"] Oct 31 12:45:04.362953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.362917 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkrr\" (UniqueName: \"kubernetes.io/projected/995e8461-e024-4e86-899d-b5cf67abb1e0-kube-api-access-2fkrr\") pod \"payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.363225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.363203 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8461-e024-4e86-899d-b5cf67abb1e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.383236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.383206 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-8faad6\"" Oct 31 12:45:04.426654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.426457 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:04.481486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.466520 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8461-e024-4e86-899d-b5cf67abb1e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.481486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.466759 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkrr\" (UniqueName: \"kubernetes.io/projected/995e8461-e024-4e86-899d-b5cf67abb1e0-kube-api-access-2fkrr\") pod \"payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.481486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.468074 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8461-e024-4e86-899d-b5cf67abb1e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.501444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.501414 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkrr\" (UniqueName: \"kubernetes.io/projected/995e8461-e024-4e86-899d-b5cf67abb1e0-kube-api-access-2fkrr\") pod \"payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.617909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.617872 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:04.950850 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:45:04.950815 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995e8461_e024_4e86_899d_b5cf67abb1e0.slice/crio-d6e4105c99b1d3e70e6f02a9a21cb4ec1a17a97093ac91c16be950c225348c5f WatchSource:0}: Error finding container d6e4105c99b1d3e70e6f02a9a21cb4ec1a17a97093ac91c16be950c225348c5f: Status 404 returned error can't find the container with id d6e4105c99b1d3e70e6f02a9a21cb4ec1a17a97093ac91c16be950c225348c5f Oct 31 12:45:04.951528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:04.951496 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll"] Oct 31 12:45:05.296352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:05.296281 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:05.924927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:05.924889 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" event={"ID":"995e8461-e024-4e86-899d-b5cf67abb1e0","Type":"ContainerStarted","Data":"b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273"} Oct 31 12:45:05.925120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:05.924950 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" event={"ID":"995e8461-e024-4e86-899d-b5cf67abb1e0","Type":"ContainerStarted","Data":"d6e4105c99b1d3e70e6f02a9a21cb4ec1a17a97093ac91c16be950c225348c5f"} Oct 31 12:45:05.925826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:05.925792 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:05.955784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:05.955732 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" podStartSLOduration=1.955718088 podStartE2EDuration="1.955718088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:05.952684756 +0000 UTC m=+1016.767699142" watchObservedRunningTime="2026-10-31 12:45:05.955718088 +0000 UTC m=+1016.770732480" Oct 31 12:45:06.007091 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:06.007062 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:06.147393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:06.147254 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:06.417126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:06.417089 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:06.653140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:06.653104 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:45:07.229656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:07.229629 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:07.367152 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:07.367118 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:07.649693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:07.649665 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:07.785190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:07.785160 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:08.674486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:08.670886 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:09.559604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:09.559551 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:10.945231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:10.945197 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:10.961041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:10.959897 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:10.994596 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:10.994550 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.031556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.030564 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:45:11.031902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.031816 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.076980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.076927 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:45:11.078270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.077229 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.087522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.087490 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.170796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.168084 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:45:11.185573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.185534 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.205644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.205558 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:45:11.205885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.205852 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.222194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.222164 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.449486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.445226 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:45:11.663591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.653539 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:45:11.960427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.960274 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:11.961790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.961758 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:45:11.962548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.962527 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:45:12.014809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:11.990019 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:45:12.158448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:12.158418 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:12.854643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:12.854610 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:13.752258 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:13.752225 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:14.346237 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.346203 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp"] Oct 31 12:45:14.355614 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.355583 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.359861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.359835 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-8faad6-serving-cert\"" Oct 31 12:45:14.363140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.363115 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp"] Oct 31 12:45:14.448658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.448630 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:14.455280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.455255 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.455501 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.455483 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab740d09-6da9-4783-b693-8e6d88b8b5ee-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.455558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.455509 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8gw\" (UniqueName: \"kubernetes.io/projected/ab740d09-6da9-4783-b693-8e6d88b8b5ee-kube-api-access-kc8gw\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.455813 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.455796 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.557642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.557605 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab740d09-6da9-4783-b693-8e6d88b8b5ee-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.557642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.557637 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8gw\" (UniqueName: \"kubernetes.io/projected/ab740d09-6da9-4783-b693-8e6d88b8b5ee-kube-api-access-kc8gw\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.557908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.557858 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.557908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.557901 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.558289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.558257 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab740d09-6da9-4783-b693-8e6d88b8b5ee-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.560247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.560208 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.560411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.560392 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.569383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.569363 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8gw\" (UniqueName: \"kubernetes.io/projected/ab740d09-6da9-4783-b693-8e6d88b8b5ee-kube-api-access-kc8gw\") pod \"maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.594917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.594887 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:14.677113 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.677012 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:14.934656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.934550 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:14.975359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:14.975329 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp"] Oct 31 12:45:14.975727 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:45:14.975704 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab740d09_6da9_4783_b693_8e6d88b8b5ee.slice/crio-542648d0caa1fb4e60034ad1dcefc4bc560a6d3041d4e407fae4012d5db5259a WatchSource:0}: Error finding container 542648d0caa1fb4e60034ad1dcefc4bc560a6d3041d4e407fae4012d5db5259a: Status 404 returned error can't find the container with id 542648d0caa1fb4e60034ad1dcefc4bc560a6d3041d4e407fae4012d5db5259a Oct 31 12:45:15.281168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:15.281077 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" event={"ID":"ab740d09-6da9-4783-b693-8e6d88b8b5ee","Type":"ContainerStarted","Data":"542648d0caa1fb4e60034ad1dcefc4bc560a6d3041d4e407fae4012d5db5259a"} Oct 31 12:45:15.403028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:15.402997 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:16.135171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:16.135136 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:16.290548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:16.290507 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" event={"ID":"ab740d09-6da9-4783-b693-8e6d88b8b5ee","Type":"ContainerStarted","Data":"32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012"} Oct 31 12:45:16.290712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:16.290663 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:16.313735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:16.313693 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" podStartSLOduration=1.820011152 podStartE2EDuration="2.313680962s" totalImagesPullingTime="493.66981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:16.312234873 +0000 UTC m=+1027.127249264" watchObservedRunningTime="2026-10-31 12:45:16.313680962 +0000 UTC m=+1027.128695353" Oct 31 12:45:16.785701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:16.785670 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:17.637395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.637357 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm"] Oct 31 12:45:17.643667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.643644 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.655203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.651021 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e1fc0ffe-serving-cert\"" Oct 31 12:45:17.673276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.673248 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm"] Oct 31 12:45:17.731375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.731340 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb660f2-f9ad-4e11-86af-51690fc91f24-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.731554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.731499 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxgnm\" (UniqueName: \"kubernetes.io/projected/feb660f2-f9ad-4e11-86af-51690fc91f24-kube-api-access-hxgnm\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.731637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.731570 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-metrics-certs\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.731704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.731668 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.733664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.733641 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:17.833685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.833642 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb660f2-f9ad-4e11-86af-51690fc91f24-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.833890 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.833738 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxgnm\" (UniqueName: \"kubernetes.io/projected/feb660f2-f9ad-4e11-86af-51690fc91f24-kube-api-access-hxgnm\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.833890 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.833798 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-metrics-certs\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.833890 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.833843 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.834087 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:45:17.833998 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-serving-cert: secret "maas-api-e2e-mt-e1fc0ffe-serving-cert" not found Oct 31 12:45:17.834087 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:45:17.834079 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls podName:feb660f2-f9ad-4e11-86af-51690fc91f24 nodeName:}" failed. No retries permitted until 2026-10-31 12:45:18.334060613 +0000 UTC m=+1029.149074999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls") pod "maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" (UID: "feb660f2-f9ad-4e11-86af-51690fc91f24") : secret "maas-api-e2e-mt-e1fc0ffe-serving-cert" not found Oct 31 12:45:17.834450 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.834425 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb660f2-f9ad-4e11-86af-51690fc91f24-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.836937 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.836898 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-metrics-certs\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.847511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.847451 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxgnm\" (UniqueName: \"kubernetes.io/projected/feb660f2-f9ad-4e11-86af-51690fc91f24-kube-api-access-hxgnm\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:17.872387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.872356 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:17.955806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:17.955725 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 12:45:18.173617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:18.173586 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:18.343495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:18.342287 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:18.345716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:18.345691 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:18.572226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:18.572192 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:18.764070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:18.764035 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:18.882330 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:18.882301 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm"] Oct 31 12:45:18.892052 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:45:18.892014 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb660f2_f9ad_4e11_86af_51690fc91f24.slice/crio-409ec1b31ebce3d64d4e32da40549cf8bf917c30c075177e0e8026ba6eb3b02b WatchSource:0}: Error finding container 409ec1b31ebce3d64d4e32da40549cf8bf917c30c075177e0e8026ba6eb3b02b: Status 404 returned error can't find the container with id 409ec1b31ebce3d64d4e32da40549cf8bf917c30c075177e0e8026ba6eb3b02b Oct 31 12:45:19.316268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:19.316239 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" event={"ID":"feb660f2-f9ad-4e11-86af-51690fc91f24","Type":"ContainerStarted","Data":"409ec1b31ebce3d64d4e32da40549cf8bf917c30c075177e0e8026ba6eb3b02b"} Oct 31 12:45:19.631024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:19.630977 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:20.326302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:20.326268 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" event={"ID":"feb660f2-f9ad-4e11-86af-51690fc91f24","Type":"ContainerStarted","Data":"6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa"} Oct 31 12:45:20.326749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:20.326476 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:20.351728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:20.351685 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" podStartSLOduration=2.967539809 podStartE2EDuration="3.351673447s" totalImagesPullingTime="384.133638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:20.348364043 +0000 UTC m=+1031.163378434" watchObservedRunningTime="2026-10-31 12:45:20.351673447 +0000 UTC m=+1031.166687837" Oct 31 12:45:20.448087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:20.448061 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:21.325331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:21.325293 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:22.197316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:22.197074 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:22.305347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:22.305312 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 12:45:22.343187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:22.343157 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:22.482036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:22.481946 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:22.878539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:22.878508 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:23.631795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:23.631767 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:24.329575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:24.329542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:24.980875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:24.980840 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:25.265440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:25.265364 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:25.399626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:25.399596 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:25.648840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:25.648802 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:26.415485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:26.411613 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:45:26.753746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:26.753661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:27.620824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:27.620792 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:28.349276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:28.349240 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:28.599454 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:28.599323 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:29.372905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:29.372874 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:29.506725 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:29.506672 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:30.130123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:30.130090 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:30.296401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:30.296359 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:30.453720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:30.453632 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:30.509027 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:45:30.508983 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:45:31.162637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:31.162597 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:31.318084 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:31.318047 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:31.460777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:31.460691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:32.331796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:32.331767 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:33.011287 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:33.011251 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:33.791986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:33.791949 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:34.069252 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:34.069168 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:34.844811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:34.844760 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:35.567922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:35.567884 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:36.120714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:36.120679 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b"] Oct 31 12:45:36.222886 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:36.222854 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:36.350876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:36.350840 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:37.012142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:37.012094 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:37.791840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:37.791807 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:38.649104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:38.649068 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:39.407395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:39.407339 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:40.085209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:40.085158 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:40.822632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:40.822595 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:41.121104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.121013 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" podUID="a47a8e05-50ff-4398-b60c-24f78bdc9c84" containerName="payload-pre-processing" containerID="cri-o://e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d" gracePeriod=25 Oct 31 12:45:41.437084 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.437056 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:45:41.580656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.580626 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47a8e05-50ff-4398-b60c-24f78bdc9c84-kube-api-access-h4wtd\" (UniqueName: \"kubernetes.io/projected/a47a8e05-50ff-4398-b60c-24f78bdc9c84-kube-api-access-h4wtd\") pod \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " Oct 31 12:45:41.580848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.580729 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47a8e05-50ff-4398-b60c-24f78bdc9c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a8e05-50ff-4398-b60c-24f78bdc9c84-plugins-config-volume\") pod \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\" (UID: \"a47a8e05-50ff-4398-b60c-24f78bdc9c84\") " Oct 31 12:45:41.581200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.581168 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47a8e05-50ff-4398-b60c-24f78bdc9c84-plugins-config-volume" pod "a47a8e05-50ff-4398-b60c-24f78bdc9c84" (UID: "a47a8e05-50ff-4398-b60c-24f78bdc9c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:45:41.581580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.581545 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a8e05-50ff-4398-b60c-24f78bdc9c84-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:41.582946 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.582920 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47a8e05-50ff-4398-b60c-24f78bdc9c84-kube-api-access-h4wtd" pod "a47a8e05-50ff-4398-b60c-24f78bdc9c84" (UID: "a47a8e05-50ff-4398-b60c-24f78bdc9c84"). InnerVolumeSpecName "kube-api-access-h4wtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:45:41.615829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.615801 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:41.684527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.684373 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wtd\" (UniqueName: \"kubernetes.io/projected/a47a8e05-50ff-4398-b60c-24f78bdc9c84-kube-api-access-h4wtd\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:41.706340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.706287 6727 generic.go:356] "Generic (PLEG): container finished" podID="a47a8e05-50ff-4398-b60c-24f78bdc9c84" containerID="e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d" exitCode=0 Oct 31 12:45:41.706559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.706535 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" Oct 31 12:45:41.706692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.706435 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" event={"ID":"a47a8e05-50ff-4398-b60c-24f78bdc9c84","Type":"ContainerDied","Data":"e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d"} Oct 31 12:45:41.706835 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.706816 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b" event={"ID":"a47a8e05-50ff-4398-b60c-24f78bdc9c84","Type":"ContainerDied","Data":"620f0e0f6d258a881e048c9c042dc0744640b423de38dc12b18c59c1864e3bbe"} Oct 31 12:45:41.706967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.706940 6727 scope.go:122] "RemoveContainer" containerID="e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d" Oct 31 12:45:41.722958 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.722930 6727 scope.go:122] "RemoveContainer" containerID="e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d" Oct 31 12:45:41.723849 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:45:41.723822 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d\": container with ID starting with e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d not found: ID does not exist" containerID="e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d" Oct 31 12:45:41.723962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.723861 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d"} err="failed to get container status \"e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d\": rpc error: code = NotFound desc = could not find container \"e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d\": container with ID starting with e0610ce8cb7456f68acd1b5979af65a0121e1144a45df3458ab4d6a609ca0a3d not found: ID does not exist" Oct 31 12:45:41.742416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.742386 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b"] Oct 31 12:45:41.746155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:41.746128 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed25a690-7845f6d79-sfw7b"] Oct 31 12:45:42.426858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:42.426818 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:42.977694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:42.977660 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb"] Oct 31 12:45:42.977952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:42.977928 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" podUID="36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" containerName="maas-api" containerID="cri-o://7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2" gracePeriod=30 Oct 31 12:45:43.064098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.064065 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:43.317687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.231432 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:43.451853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.451311 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47a8e05-50ff-4398-b60c-24f78bdc9c84" path="/var/lib/kubelet/pods/a47a8e05-50ff-4398-b60c-24f78bdc9c84/volumes" Oct 31 12:45:43.543836 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.543806 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:45:43.652036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.651998 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:43.656565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.656544 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-openshift-service-ca\") pod \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " Oct 31 12:45:43.656690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.656590 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-kube-api-access-4mwkq\" (UniqueName: \"kubernetes.io/projected/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-kube-api-access-4mwkq\") pod \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " Oct 31 12:45:43.656809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.656782 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") pod \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " Oct 31 12:45:43.656938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.656841 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-metrics-certs\") pod \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\" (UID: \"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352\") " Oct 31 12:45:43.657157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.657135 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-openshift-service-ca" pod "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" (UID: "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:45:43.657792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.657766 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:43.659014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.658986 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls" pod "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" (UID: "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:45:43.659209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.659130 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-kube-api-access-4mwkq" pod "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" (UID: "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352"). InnerVolumeSpecName "kube-api-access-4mwkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:45:43.659275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.659211 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-metrics-certs" pod "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" (UID: "36b7dc9f-c8e0-4e5b-8207-fe52c90d0352"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:45:43.728585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.728551 6727 generic.go:356] "Generic (PLEG): container finished" podID="36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" containerID="7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2" exitCode=0 Oct 31 12:45:43.728748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.728603 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" Oct 31 12:45:43.728748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.728639 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" event={"ID":"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352","Type":"ContainerDied","Data":"7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2"} Oct 31 12:45:43.728748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.728695 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb" event={"ID":"36b7dc9f-c8e0-4e5b-8207-fe52c90d0352","Type":"ContainerDied","Data":"7774183d67a413914b4f7406058862b6cc0ec40abd06f3183dfc799cf43bf069"} Oct 31 12:45:43.728748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.728722 6727 scope.go:122] "RemoveContainer" containerID="7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2" Oct 31 12:45:43.745843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.745821 6727 scope.go:122] "RemoveContainer" containerID="7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2" Oct 31 12:45:43.746182 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:45:43.746155 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2\": container with ID starting with 7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2 not found: ID does not exist" containerID="7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2" Oct 31 12:45:43.746264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.746194 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2"} err="failed to get container status \"7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2\": rpc error: code = NotFound desc = could not find container \"7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2\": container with ID starting with 7a790674d4a669927982c19a24a12ae342050df461cfed7b17701411b52840d2 not found: ID does not exist" Oct 31 12:45:43.759457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.759434 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:43.759457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.759459 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:43.759607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.759487 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwkq\" (UniqueName: \"kubernetes.io/projected/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352-kube-api-access-4mwkq\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:43.770293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.770265 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb"] Oct 31 12:45:43.775657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.775627 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed25a690-55c58cff5b-65khb"] Oct 31 12:45:43.823091 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:43.823049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:44.060642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:44.060563 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:44.517594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:44.517563 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:45.263971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:45.263940 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:45.303364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:45.303332 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" path="/var/lib/kubelet/pods/36b7dc9f-c8e0-4e5b-8207-fe52c90d0352/volumes" Oct 31 12:45:45.988606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:45.988568 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:46.222576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:46.222542 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:45:46.828692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:46.828663 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:47.080242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:47.080164 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:47.527430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:47.527311 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:47.676337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:47.676301 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:47.970987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:47.970955 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:48.913216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:48.911961 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:49.813515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:49.810780 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:50.280489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:50.280066 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:50.657950 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:50.657910 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:51.271975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.271909 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:51.527544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.480440 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:51.699487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.699262 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:51.774537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.771022 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj"] Oct 31 12:45:51.774537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.771381 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" containerID="cri-o://f2460625c80c469fc7dc05c562aaa4a8f6d7df5b55a463ba2a9f8d8d820980dd" gracePeriod=30 Oct 31 12:45:51.904566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.904522 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j"] Oct 31 12:45:51.932846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.932786 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" containerID="cri-o://fd8a5f296ae2279dc1d02ab83a5fa9a96c9a7e1cf80c0e73506efa5be3485916" gracePeriod=30 Oct 31 12:45:51.989611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:51.989568 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:52.107542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.085850 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:52.117591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.117548 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:52.138772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.138731 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:45:52.246580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.180198 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:52.246580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.186276 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:52.246580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.187747 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:52.246580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.206625 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:45:52.391572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.377930 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:52.524488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.521188 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:45:52.598043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.597986 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:52.752844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.752755 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:52.904153 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:52.904120 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:53.893105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:53.893074 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:54.546823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.546786 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:54.613554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.613528 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf"] Oct 31 12:45:54.614320 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.614305 6727 cpu_manager.go:414] "containerMap: removing container" podUID="a47a8e05-50ff-4398-b60c-24f78bdc9c84" containerName="payload-pre-processing" Oct 31 12:45:54.614374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.614321 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47a8e05-50ff-4398-b60c-24f78bdc9c84" containerName="payload-pre-processing" Oct 31 12:45:54.614374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.614357 6727 cpu_manager.go:414] "containerMap: removing container" podUID="36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" containerName="maas-api" Oct 31 12:45:54.614374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.614364 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" containerName="maas-api" Oct 31 12:45:54.614532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.614483 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b7dc9f-c8e0-4e5b-8207-fe52c90d0352" containerName="maas-api" Oct 31 12:45:54.614532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.614505 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47a8e05-50ff-4398-b60c-24f78bdc9c84" containerName="payload-pre-processing" Oct 31 12:45:54.617618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.617600 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:54.627057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.626080 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5d310d-dockercfg-z477c\"" Oct 31 12:45:54.627057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.626118 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5d310d\"" Oct 31 12:45:54.634993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.634966 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf"] Oct 31 12:45:54.647565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.647540 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh"] Oct 31 12:45:54.655424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.655396 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:54.665562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.665499 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh"] Oct 31 12:45:54.684068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.684040 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj4nx\" (UniqueName: \"kubernetes.io/projected/302a3e61-bc0d-485f-8a31-36b4111b2d36-kube-api-access-gj4nx\") pod \"payload-processing-e2e-api-a-5d310d-7889d55669-smsmf\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:54.687494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.684654 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a3e61-bc0d-485f-8a31-36b4111b2d36-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d310d-7889d55669-smsmf\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:54.708418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.708374 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:54.786653 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.786597 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66786b56-88d9-4568-b78e-0835cb1d1fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:54.786802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.786765 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a3e61-bc0d-485f-8a31-36b4111b2d36-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d310d-7889d55669-smsmf\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:54.786961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.786946 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqvx9\" (UniqueName: \"kubernetes.io/projected/66786b56-88d9-4568-b78e-0835cb1d1fab-kube-api-access-nqvx9\") pod \"payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:54.787036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.786984 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj4nx\" (UniqueName: \"kubernetes.io/projected/302a3e61-bc0d-485f-8a31-36b4111b2d36-kube-api-access-gj4nx\") pod \"payload-processing-e2e-api-a-5d310d-7889d55669-smsmf\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:54.787685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.787650 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a3e61-bc0d-485f-8a31-36b4111b2d36-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d310d-7889d55669-smsmf\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:54.875486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.869222 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj4nx\" (UniqueName: \"kubernetes.io/projected/302a3e61-bc0d-485f-8a31-36b4111b2d36-kube-api-access-gj4nx\") pod \"payload-processing-e2e-api-a-5d310d-7889d55669-smsmf\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:54.876171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.876137 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:54.891930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.891891 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqvx9\" (UniqueName: \"kubernetes.io/projected/66786b56-88d9-4568-b78e-0835cb1d1fab-kube-api-access-nqvx9\") pod \"payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:54.892350 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.892274 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66786b56-88d9-4568-b78e-0835cb1d1fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:54.900489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:54.893568 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66786b56-88d9-4568-b78e-0835cb1d1fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:55.020483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:55.004958 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:55.059734 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:55.054566 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqvx9\" (UniqueName: \"kubernetes.io/projected/66786b56-88d9-4568-b78e-0835cb1d1fab-kube-api-access-nqvx9\") pod \"payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:55.288036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:55.288005 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:55.418004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:55.417345 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf"] Oct 31 12:45:55.419360 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:45:55.418434 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302a3e61_bc0d_485f_8a31_36b4111b2d36.slice/crio-b6f08b1dc98c39c69b9447d4c4ad2c0befbf9cf50ac2704519970415d02cf388 WatchSource:0}: Error finding container b6f08b1dc98c39c69b9447d4c4ad2c0befbf9cf50ac2704519970415d02cf388: Status 404 returned error can't find the container with id b6f08b1dc98c39c69b9447d4c4ad2c0befbf9cf50ac2704519970415d02cf388 Oct 31 12:45:55.437547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:55.437453 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" event={"ID":"302a3e61-bc0d-485f-8a31-36b4111b2d36","Type":"ContainerStarted","Data":"b6f08b1dc98c39c69b9447d4c4ad2c0befbf9cf50ac2704519970415d02cf388"} Oct 31 12:45:55.583966 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:45:55.583931 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66786b56_88d9_4568_b78e_0835cb1d1fab.slice/crio-4fffea1118987128a0d7512f94f8954cd417556795ad608063353ef8bdbac7af WatchSource:0}: Error finding container 4fffea1118987128a0d7512f94f8954cd417556795ad608063353ef8bdbac7af: Status 404 returned error can't find the container with id 4fffea1118987128a0d7512f94f8954cd417556795ad608063353ef8bdbac7af Oct 31 12:45:55.584911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:55.584884 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh"] Oct 31 12:45:55.737697 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:55.737623 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:56.448603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.448567 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" event={"ID":"302a3e61-bc0d-485f-8a31-36b4111b2d36","Type":"ContainerStarted","Data":"892d9ad6f252ee9165a95eb54b87f9326a402f4c51f0d1dbff80f0f2ef5dbc6b"} Oct 31 12:45:56.449089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.448659 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:45:56.451151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.451121 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" event={"ID":"66786b56-88d9-4568-b78e-0835cb1d1fab","Type":"ContainerStarted","Data":"2694633dc55f1e8724c24cef352a38249af589567703a409ba57292acb958006"} Oct 31 12:45:56.451299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.451170 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" event={"ID":"66786b56-88d9-4568-b78e-0835cb1d1fab","Type":"ContainerStarted","Data":"4fffea1118987128a0d7512f94f8954cd417556795ad608063353ef8bdbac7af"} Oct 31 12:45:56.451299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.451235 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:45:56.475144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.475107 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:56.499059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.499001 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" podStartSLOduration=2.498985633 podStartE2EDuration="2.498985633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:56.495176786 +0000 UTC m=+1067.310191191" watchObservedRunningTime="2026-10-31 12:45:56.498985633 +0000 UTC m=+1067.314000052" Oct 31 12:45:56.499493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:56.499429 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" podStartSLOduration=2.499419785 podStartE2EDuration="2.499419785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:56.469410455 +0000 UTC m=+1067.284424860" watchObservedRunningTime="2026-10-31 12:45:56.499419785 +0000 UTC m=+1067.314434178" Oct 31 12:45:57.077910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.077876 6727 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:45:57.078103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.077923 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:45:57.274863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.274706 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:57.413214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.413174 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:57.468236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.468189 6727 generic.go:356] "Generic (PLEG): container finished" podID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerID="fd8a5f296ae2279dc1d02ab83a5fa9a96c9a7e1cf80c0e73506efa5be3485916" exitCode=0 Oct 31 12:45:57.468842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.468381 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" event={"ID":"d7b282d2-95dc-4f43-84c6-77271c02e92e","Type":"ContainerDied","Data":"fd8a5f296ae2279dc1d02ab83a5fa9a96c9a7e1cf80c0e73506efa5be3485916"} Oct 31 12:45:57.468842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.468479 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" event={"ID":"d7b282d2-95dc-4f43-84c6-77271c02e92e","Type":"ContainerDied","Data":"ccb4555464b1e37911f97b1476a7ff3122f093037de296b312241194cf01396a"} Oct 31 12:45:57.468842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.468507 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb4555464b1e37911f97b1476a7ff3122f093037de296b312241194cf01396a" Oct 31 12:45:57.469226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.469209 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:45:57.470672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.470651 6727 generic.go:356] "Generic (PLEG): container finished" podID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerID="f2460625c80c469fc7dc05c562aaa4a8f6d7df5b55a463ba2a9f8d8d820980dd" exitCode=0 Oct 31 12:45:57.471840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.471820 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" event={"ID":"79b6ad2e-85f2-4148-a9a8-d7c53d72d286","Type":"ContainerDied","Data":"f2460625c80c469fc7dc05c562aaa4a8f6d7df5b55a463ba2a9f8d8d820980dd"} Oct 31 12:45:57.547150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.547125 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:45:57.585104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585072 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-token\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-token\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585315 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585298 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b282d2-95dc-4f43-84c6-77271c02e92e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b282d2-95dc-4f43-84c6-77271c02e92e-istiod-ca-cert\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585361 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-kube-api-access-shbb8\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-kube-api-access-shbb8\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585475 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585449 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-data\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585536 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-socket\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585581 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-envoy\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585647 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-podinfo\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585761 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-credential-socket\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.585843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585790 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b282d2-95dc-4f43-84c6-77271c02e92e-istiod-ca-cert" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:45:57.585904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585862 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-certs\") pod \"d7b282d2-95dc-4f43-84c6-77271c02e92e\" (UID: \"d7b282d2-95dc-4f43-84c6-77271c02e92e\") " Oct 31 12:45:57.586014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.585983 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-socket" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.586243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.586213 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-credential-socket" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.586339 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.586320 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-data" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.586578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.586557 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-certs" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.588067 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.588045 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-token" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:45:57.588160 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.588087 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-envoy" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.588699 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.588668 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-kube-api-access-shbb8" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "kube-api-access-shbb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:45:57.589384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.589366 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-podinfo" pod "d7b282d2-95dc-4f43-84c6-77271c02e92e" (UID: "d7b282d2-95dc-4f43-84c6-77271c02e92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:45:57.594204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594149 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b282d2-95dc-4f43-84c6-77271c02e92e-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594171 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shbb8\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-kube-api-access-shbb8\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594182 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594195 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594208 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594217 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594225 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594234 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b282d2-95dc-4f43-84c6-77271c02e92e-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.594399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.594244 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b282d2-95dc-4f43-84c6-77271c02e92e-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.695256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695213 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-credential-socket\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695303 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-data\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695330 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-certs\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695351 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-envoy\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695400 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-socket\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695450 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-token\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-token\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695452 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-credential-socket" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.695663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695609 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-data" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.695809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695658 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-certs" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.695809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695699 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-podinfo\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695757 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istiod-ca-cert\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.695809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695756 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-socket" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.696018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.695870 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-kube-api-access-64xg8\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-kube-api-access-64xg8\") pod \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\" (UID: \"79b6ad2e-85f2-4148-a9a8-d7c53d72d286\") " Oct 31 12:45:57.696112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.696079 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istiod-ca-cert" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:45:57.696613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.696592 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.696613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.696614 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.696779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.696623 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.696779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.696632 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.696779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.696641 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.697683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.697652 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-token" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:45:57.697803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.697767 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-envoy" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:45:57.698041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.698024 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-podinfo" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:45:57.698126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.698110 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-kube-api-access-64xg8" pod "79b6ad2e-85f2-4148-a9a8-d7c53d72d286" (UID: "79b6ad2e-85f2-4148-a9a8-d7c53d72d286"). InnerVolumeSpecName "kube-api-access-64xg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:45:57.798030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.797992 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.798030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.798027 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64xg8\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-kube-api-access-64xg8\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.798030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.798038 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:57.798251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:57.798047 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6ad2e-85f2-4148-a9a8-d7c53d72d286-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:45:58.018265 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.018181 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:58.481371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.481333 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" event={"ID":"79b6ad2e-85f2-4148-a9a8-d7c53d72d286","Type":"ContainerDied","Data":"b4eb4adfe7bce864c1e9b5cce1db3391fa187f431ac27a2739a2421e23da41e0"} Oct 31 12:45:58.481371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.481354 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j" Oct 31 12:45:58.481916 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.481397 6727 scope.go:122] "RemoveContainer" containerID="f2460625c80c469fc7dc05c562aaa4a8f6d7df5b55a463ba2a9f8d8d820980dd" Oct 31 12:45:58.481916 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.481352 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj" Oct 31 12:45:58.510265 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.510238 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj"] Oct 31 12:45:58.516422 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.516402 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-2b7rj"] Oct 31 12:45:58.528684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.528662 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j"] Oct 31 12:45:58.532285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.532264 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-7vq5j"] Oct 31 12:45:58.779720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:58.779649 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:59.305830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:59.305803 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" path="/var/lib/kubelet/pods/79b6ad2e-85f2-4148-a9a8-d7c53d72d286/volumes" Oct 31 12:45:59.306359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:59.306344 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" path="/var/lib/kubelet/pods/d7b282d2-95dc-4f43-84c6-77271c02e92e/volumes" Oct 31 12:45:59.522048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:59.522014 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:59.661777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:59.661750 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:59.965669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:45:59.965596 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:00.235991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:00.235918 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:00.437548 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:46:00.437009 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:46:00.534872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:00.534793 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:01.128404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:01.128370 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:01.266178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:01.266145 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:01.932351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:01.932315 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:02.651029 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:02.650997 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:03.434217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.431997 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5"] Oct 31 12:46:03.434217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.433127 6727 cpu_manager.go:414] "containerMap: removing container" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" Oct 31 12:46:03.434217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.433144 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" Oct 31 12:46:03.434217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.433197 6727 cpu_manager.go:414] "containerMap: removing container" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" Oct 31 12:46:03.434217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.433202 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" Oct 31 12:46:03.434217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.433312 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b6ad2e-85f2-4148-a9a8-d7c53d72d286" containerName="istio-proxy" Oct 31 12:46:03.434217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.433325 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b282d2-95dc-4f43-84c6-77271c02e92e" containerName="istio-proxy" Oct 31 12:46:03.439823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.439374 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.457480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.449958 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5"] Oct 31 12:46:03.457480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.454708 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5d310d-serving-cert\"" Oct 31 12:46:03.506884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.506852 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81df80ca-28f7-465a-a4c6-09bbbea0b503-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.507091 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.507068 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-metrics-certs\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.507281 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.507248 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fc9h\" (UniqueName: \"kubernetes.io/projected/81df80ca-28f7-465a-a4c6-09bbbea0b503-kube-api-access-4fc9h\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.507419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.507395 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-maas-api-tls\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.526137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.526110 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:03.609021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.608996 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-metrics-certs\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.609154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.609083 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fc9h\" (UniqueName: \"kubernetes.io/projected/81df80ca-28f7-465a-a4c6-09bbbea0b503-kube-api-access-4fc9h\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.609154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.609140 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-maas-api-tls\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.609291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.609271 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81df80ca-28f7-465a-a4c6-09bbbea0b503-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.609858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.609834 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81df80ca-28f7-465a-a4c6-09bbbea0b503-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.611458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.611430 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-metrics-certs\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.611666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.611649 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-maas-api-tls\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.620734 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.620714 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fc9h\" (UniqueName: \"kubernetes.io/projected/81df80ca-28f7-465a-a4c6-09bbbea0b503-kube-api-access-4fc9h\") pod \"maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.670672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.670646 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:03.782641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.782335 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:03.913928 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:03.911162 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:04.135051 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:04.127381 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:04.199784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:04.187334 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5"] Oct 31 12:46:04.199784 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:46:04.187673 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81df80ca_28f7_465a_a4c6_09bbbea0b503.slice/crio-f1cf983d3ae02a6b9a7b9f0c88ebde9c7ad79369e9a315f7e5c2f89ef4f8cb9f WatchSource:0}: Error finding container f1cf983d3ae02a6b9a7b9f0c88ebde9c7ad79369e9a315f7e5c2f89ef4f8cb9f: Status 404 returned error can't find the container with id f1cf983d3ae02a6b9a7b9f0c88ebde9c7ad79369e9a315f7e5c2f89ef4f8cb9f Oct 31 12:46:04.283309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:04.283275 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:04.549427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:04.549333 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" event={"ID":"81df80ca-28f7-465a-a4c6-09bbbea0b503","Type":"ContainerStarted","Data":"f1cf983d3ae02a6b9a7b9f0c88ebde9c7ad79369e9a315f7e5c2f89ef4f8cb9f"} Oct 31 12:46:04.891491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:04.891369 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:05.527541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:05.527504 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:05.559883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:05.559846 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" event={"ID":"81df80ca-28f7-465a-a4c6-09bbbea0b503","Type":"ContainerStarted","Data":"e11655b442aff96db2acadd0c681ed6772fe4b5b2d166832aa11cdb6164c892d"} Oct 31 12:46:05.560249 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:05.560081 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:05.594547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:05.594487 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" podStartSLOduration=2.130773431 podStartE2EDuration="2.594453058s" totalImagesPullingTime="463.679627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:05.591780504 +0000 UTC m=+1076.406794910" watchObservedRunningTime="2026-10-31 12:46:05.594453058 +0000 UTC m=+1076.409467450" Oct 31 12:46:05.700138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:05.700105 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:05.850486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:05.849917 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:06.619048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:06.619015 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:07.226401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:07.226369 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:07.451502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:07.451452 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:07.480456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:07.480333 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:46:07.481962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:07.481929 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:46:08.269220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:08.269170 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:08.934213 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:08.934158 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:09.349694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:09.349663 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:09.479628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:09.479598 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:09.724666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:09.724581 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:10.343179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:10.343147 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:11.107332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:11.107300 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:11.573968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:11.573943 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:46:11.924159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:11.924075 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:13.395155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:13.395119 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:14.018822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:14.018789 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:14.819232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:14.819198 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:14.963254 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:14.963218 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:15.763909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:15.763868 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:16.485113 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:16.485078 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:17.421413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:17.421368 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:17.738941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:17.738857 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:18.086384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:18.086345 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:18.374069 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:18.373979 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:18.761410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:18.761312 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:18.934484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:18.933976 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:19.649791 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:19.649716 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:19.791801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:19.791772 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:20.420017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:20.419983 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:21.218529 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:21.218495 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:22.020684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:22.020642 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:22.723510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:22.723480 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:22.898079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:22.898043 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:23.793663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:23.793634 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:24.019942 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.019913 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:24.054817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.054749 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz"] Oct 31 12:46:24.059927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.059900 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.066781 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.066740 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-711af2-dockercfg-xrqtf\"" Oct 31 12:46:24.069746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.069721 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-711af2\"" Oct 31 12:46:24.096933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.096894 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz"] Oct 31 12:46:24.107332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.107303 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z"] Oct 31 12:46:24.112163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.112141 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.126839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.126813 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z"] Oct 31 12:46:24.129917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.129888 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c86f73-1646-45bc-a5b7-65d623f3aea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.130355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.130321 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20744874-84f3-4fc0-befd-26dc76141b93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.130632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.130596 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9vp\" (UniqueName: \"kubernetes.io/projected/20744874-84f3-4fc0-befd-26dc76141b93-kube-api-access-rl9vp\") pod \"payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.130791 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.130773 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh28j\" (UniqueName: \"kubernetes.io/projected/00c86f73-1646-45bc-a5b7-65d623f3aea6-kube-api-access-qh28j\") pod \"payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.212486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.199307 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:24.266517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.235618 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20744874-84f3-4fc0-befd-26dc76141b93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.355518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.348710 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9vp\" (UniqueName: \"kubernetes.io/projected/20744874-84f3-4fc0-befd-26dc76141b93-kube-api-access-rl9vp\") pod \"payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.355518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.348767 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh28j\" (UniqueName: \"kubernetes.io/projected/00c86f73-1646-45bc-a5b7-65d623f3aea6-kube-api-access-qh28j\") pod \"payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.355518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.348990 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c86f73-1646-45bc-a5b7-65d623f3aea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.355518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.350131 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c86f73-1646-45bc-a5b7-65d623f3aea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.355518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.283044 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20744874-84f3-4fc0-befd-26dc76141b93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.397102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.388425 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh28j\" (UniqueName: \"kubernetes.io/projected/00c86f73-1646-45bc-a5b7-65d623f3aea6-kube-api-access-qh28j\") pod \"payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.397102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.390085 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9vp\" (UniqueName: \"kubernetes.io/projected/20744874-84f3-4fc0-befd-26dc76141b93-kube-api-access-rl9vp\") pod \"payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.397796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.397766 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:24.442272 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.439993 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:24.726514 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:46:24.726479 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c86f73_1646_45bc_a5b7_65d623f3aea6.slice/crio-a4921431395539658dfd8613912393695b712eaf23674df5b196e96466ff1c5f WatchSource:0}: Error finding container a4921431395539658dfd8613912393695b712eaf23674df5b196e96466ff1c5f: Status 404 returned error can't find the container with id a4921431395539658dfd8613912393695b712eaf23674df5b196e96466ff1c5f Oct 31 12:46:24.726829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.726808 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz"] Oct 31 12:46:24.748286 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.748264 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" event={"ID":"00c86f73-1646-45bc-a5b7-65d623f3aea6","Type":"ContainerStarted","Data":"a4921431395539658dfd8613912393695b712eaf23674df5b196e96466ff1c5f"} Oct 31 12:46:24.777915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.777893 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z"] Oct 31 12:46:24.778981 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:46:24.778956 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20744874_84f3_4fc0_befd_26dc76141b93.slice/crio-977d5d18ad7de6826a17782e43f1152605c5c17e462b5b3a4a2c171213605e58 WatchSource:0}: Error finding container 977d5d18ad7de6826a17782e43f1152605c5c17e462b5b3a4a2c171213605e58: Status 404 returned error can't find the container with id 977d5d18ad7de6826a17782e43f1152605c5c17e462b5b3a4a2c171213605e58 Oct 31 12:46:24.918444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:24.918416 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:25.759785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.759743 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" event={"ID":"20744874-84f3-4fc0-befd-26dc76141b93","Type":"ContainerStarted","Data":"fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6"} Oct 31 12:46:25.759968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.759814 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" event={"ID":"20744874-84f3-4fc0-befd-26dc76141b93","Type":"ContainerStarted","Data":"977d5d18ad7de6826a17782e43f1152605c5c17e462b5b3a4a2c171213605e58"} Oct 31 12:46:25.759968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.759844 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:25.761630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.761598 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" event={"ID":"00c86f73-1646-45bc-a5b7-65d623f3aea6","Type":"ContainerStarted","Data":"17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36"} Oct 31 12:46:25.761770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.761652 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:25.781415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.781369 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" podStartSLOduration=1.7813549709999998 podStartE2EDuration="1.781354971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:25.779358966 +0000 UTC m=+1096.594373358" watchObservedRunningTime="2026-10-31 12:46:25.781354971 +0000 UTC m=+1096.596369361" Oct 31 12:46:25.801396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.801343 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" podStartSLOduration=1.801328159 podStartE2EDuration="1.801328159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:25.800116798 +0000 UTC m=+1096.615131189" watchObservedRunningTime="2026-10-31 12:46:25.801328159 +0000 UTC m=+1096.616342551" Oct 31 12:46:25.866374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:25.866344 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:26.823709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:26.823664 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:27.730487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:27.728868 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:29.036988 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:29.036952 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:29.395042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:29.393310 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:30.019342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.018544 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.070990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.036571 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.073000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.072958 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.074026 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.073979 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.087156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.087117 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:30.088038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.088003 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.420671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.420593 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:30.516490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.455721 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.672547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.653577 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.732123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.732068 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.757580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.757540 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.955574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.760723 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:30.985930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:30.985869 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:31.014921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.014583 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:31.039299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.025048 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:46:31.049496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.039674 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:46:31.049496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.043275 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:31.065535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.049883 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:46:31.153758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.076511 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:31.153758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.084339 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:46:31.153758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.085379 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:46:31.153758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.086057 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:46:31.179885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.168911 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:31.201142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.201042 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:46:31.203490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.203353 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 31 12:46:31.368180 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.340406 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:31.435486 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:46:31.422690 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:46:31.572494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.556095 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:31.690753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.690668 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:46:31.696510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.696486 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:46:31.696792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.696776 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:46:31.699977 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:31.699958 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:46:32.320019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:32.319982 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:32.987695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:32.987657 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:33.812162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:33.812126 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:34.592371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:34.592312 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:35.427081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:35.427050 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:36.081694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:36.081659 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:36.226782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:36.226741 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:36.365309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:36.365240 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:36.775633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:36.775545 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:46:36.776085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:36.775699 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:46:37.028242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.028163 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf"] Oct 31 12:46:37.035076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.035053 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.038938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.038907 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-711af2-serving-cert\"" Oct 31 12:46:37.045632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.045586 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf"] Oct 31 12:46:37.097050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.097003 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-maas-api-tls\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.097274 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.097254 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-metrics-certs\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.097341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.097309 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ea685b-ff9d-4e0d-ae04-a390e74159d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.097435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.097416 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhv8\" (UniqueName: \"kubernetes.io/projected/64ea685b-ff9d-4e0d-ae04-a390e74159d0-kube-api-access-pmhv8\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.116931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.116901 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:37.199950 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.199915 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-metrics-certs\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.200137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.199997 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ea685b-ff9d-4e0d-ae04-a390e74159d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.200137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.200086 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhv8\" (UniqueName: \"kubernetes.io/projected/64ea685b-ff9d-4e0d-ae04-a390e74159d0-kube-api-access-pmhv8\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.200338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.200227 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-maas-api-tls\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.203835 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.203806 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ea685b-ff9d-4e0d-ae04-a390e74159d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.206558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.206528 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-metrics-certs\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.208824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.208797 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-maas-api-tls\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.213622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.213596 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhv8\" (UniqueName: \"kubernetes.io/projected/64ea685b-ff9d-4e0d-ae04-a390e74159d0-kube-api-access-pmhv8\") pod \"maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.267024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.266986 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:37.354136 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.354102 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:37.618941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.618868 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:37.634799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.634771 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j"] Oct 31 12:46:37.640669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.640651 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:37.649801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.649779 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bb3cf5-dockercfg-jgjjq\"" Oct 31 12:46:37.653823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.653806 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5"] Oct 31 12:46:37.657631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.657608 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bb3cf5\"" Oct 31 12:46:37.659146 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.659123 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:37.669007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.668984 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j"] Oct 31 12:46:37.684557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.684400 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf"] Oct 31 12:46:37.684557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.684455 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5"] Oct 31 12:46:37.709986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.709960 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:37.710103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.710008 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7z89\" (UniqueName: \"kubernetes.io/projected/4a8e8ba7-c995-4568-8ab6-12b61373350a-kube-api-access-s7z89\") pod \"payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:37.710167 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.710137 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm9l8\" (UniqueName: \"kubernetes.io/projected/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-kube-api-access-cm9l8\") pod \"payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:37.710334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.710318 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8e8ba7-c995-4568-8ab6-12b61373350a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:37.773121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.773087 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:37.813097 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.813053 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8e8ba7-c995-4568-8ab6-12b61373350a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:37.813805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.813270 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:37.813805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.813322 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7z89\" (UniqueName: \"kubernetes.io/projected/4a8e8ba7-c995-4568-8ab6-12b61373350a-kube-api-access-s7z89\") pod \"payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:37.813805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.813445 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm9l8\" (UniqueName: \"kubernetes.io/projected/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-kube-api-access-cm9l8\") pod \"payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:37.816990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.816964 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:37.817814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.817789 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8e8ba7-c995-4568-8ab6-12b61373350a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:37.863786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.857604 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm9l8\" (UniqueName: \"kubernetes.io/projected/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-kube-api-access-cm9l8\") pod \"payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:37.863786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.860126 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7z89\" (UniqueName: \"kubernetes.io/projected/4a8e8ba7-c995-4568-8ab6-12b61373350a-kube-api-access-s7z89\") pod \"payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:37.982961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.982879 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:37.991481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:37.989809 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:38.009077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.009045 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:38.347685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.347557 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j"] Oct 31 12:46:38.347988 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:46:38.347894 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c87e2ff_63e2_4032_92f9_8e535fe0ce47.slice/crio-b95462af72ae53dbe51196f260a0006704ef3102dafb168fa0f5a479af6e863b WatchSource:0}: Error finding container b95462af72ae53dbe51196f260a0006704ef3102dafb168fa0f5a479af6e863b: Status 404 returned error can't find the container with id b95462af72ae53dbe51196f260a0006704ef3102dafb168fa0f5a479af6e863b Oct 31 12:46:38.404227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.403867 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5"] Oct 31 12:46:38.408116 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:46:38.408087 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8e8ba7_c995_4568_8ab6_12b61373350a.slice/crio-c382fd8ca315501fb5f83b9f402a1882acea7313ce51b63b0297b5e0f3009033 WatchSource:0}: Error finding container c382fd8ca315501fb5f83b9f402a1882acea7313ce51b63b0297b5e0f3009033: Status 404 returned error can't find the container with id c382fd8ca315501fb5f83b9f402a1882acea7313ce51b63b0297b5e0f3009033 Oct 31 12:46:38.414229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.414196 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" event={"ID":"64ea685b-ff9d-4e0d-ae04-a390e74159d0","Type":"ContainerStarted","Data":"219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940"} Oct 31 12:46:38.414313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.414258 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" event={"ID":"64ea685b-ff9d-4e0d-ae04-a390e74159d0","Type":"ContainerStarted","Data":"ccd8d65b1dcf6574960cf63816811fdb3af6f42a082692b9adcf966807f5afc5"} Oct 31 12:46:38.414383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.414314 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:38.416566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.416538 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" event={"ID":"6c87e2ff-63e2-4032-92f9-8e535fe0ce47","Type":"ContainerStarted","Data":"b95462af72ae53dbe51196f260a0006704ef3102dafb168fa0f5a479af6e863b"} Oct 31 12:46:38.445386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.445339 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" podStartSLOduration=1.9385824459999998 podStartE2EDuration="2.445325711s" totalImagesPullingTime="506.743265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:38.441313486 +0000 UTC m=+1109.256327900" watchObservedRunningTime="2026-10-31 12:46:38.445325711 +0000 UTC m=+1109.260340103" Oct 31 12:46:38.874990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:38.874950 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:39.425718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.425681 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" event={"ID":"6c87e2ff-63e2-4032-92f9-8e535fe0ce47","Type":"ContainerStarted","Data":"e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61"} Oct 31 12:46:39.425906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.425766 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:39.427408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.427379 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" event={"ID":"4a8e8ba7-c995-4568-8ab6-12b61373350a","Type":"ContainerStarted","Data":"7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51"} Oct 31 12:46:39.427548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.427427 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" event={"ID":"4a8e8ba7-c995-4568-8ab6-12b61373350a","Type":"ContainerStarted","Data":"c382fd8ca315501fb5f83b9f402a1882acea7313ce51b63b0297b5e0f3009033"} Oct 31 12:46:39.427548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.427535 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:39.443743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.443693 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" podStartSLOduration=2.443677978 podStartE2EDuration="2.443677978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:39.442587051 +0000 UTC m=+1110.257601454" watchObservedRunningTime="2026-10-31 12:46:39.443677978 +0000 UTC m=+1110.258692372" Oct 31 12:46:39.466312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.466270 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" podStartSLOduration=2.466260027 podStartE2EDuration="2.466260027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:39.4624555 +0000 UTC m=+1110.277469892" watchObservedRunningTime="2026-10-31 12:46:39.466260027 +0000 UTC m=+1110.281274461" Oct 31 12:46:39.563747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:39.563716 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:40.348240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:40.348205 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:40.934329 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:40.934292 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:41.629024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:41.628987 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:42.284834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:42.284805 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:42.934652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:42.934615 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:43.675752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:43.675718 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:44.420927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:44.420896 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:44.441358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:44.441330 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:46:45.209199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:45.209157 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:45.552691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:45.552611 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:45.973110 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:45.973070 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:46.122412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:46.122381 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:46.268995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:46.268915 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:46.900439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:46.900405 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks"] Oct 31 12:46:46.906337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:46.906313 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:46.912176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:46.912130 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1fc0ffe-openshift-default-dockercfg-2nk8m\"" Oct 31 12:46:46.914008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:46.913972 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:46.922348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:46.922313 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks"] Oct 31 12:46:47.018762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.018732 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.018914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.018778 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfxb\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-kube-api-access-kpfxb\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.018914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.018888 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.018982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.018941 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.019080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.019058 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.019120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.019089 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.019179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.019167 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.019223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.019198 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.019292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.019278 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121015 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.120980 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121015 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121015 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121059 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121080 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121127 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121327 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121362 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfxb\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-kube-api-access-kpfxb\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121427 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121503 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121432 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121641 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.121799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.121742 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.122039 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.122023 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.122126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.122104 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.123427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.123406 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.123692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.123673 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.133216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.133195 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfxb\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-kube-api-access-kpfxb\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.133778 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.133758 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.226441 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.226368 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:47.523598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.523516 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks"] Oct 31 12:46:47.535540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.535495 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:46:47.535660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.535641 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:46:47.535747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.535729 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:46:47.738846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.738812 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:47.882522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:47.882484 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:48.156222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:48.156157 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:48.536692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:48.536597 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" event={"ID":"fe1e9ab1-14f7-487e-90c8-b208d18436e6","Type":"ContainerStarted","Data":"0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d"} Oct 31 12:46:48.536692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:48.536681 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" event={"ID":"fe1e9ab1-14f7-487e-90c8-b208d18436e6","Type":"ContainerStarted","Data":"4f12d564bd3c72c1df1799e49d3878ae98b987a915ce2328f578b32a342bf521"} Oct 31 12:46:48.598032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:48.597953 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podStartSLOduration=2.597931668 podStartE2EDuration="2.597931668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:48.561397607 +0000 UTC m=+1119.376412037" watchObservedRunningTime="2026-10-31 12:46:48.597931668 +0000 UTC m=+1119.412946064" Oct 31 12:46:48.738843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:48.738811 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:49.226870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:49.226836 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:49.228471 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:49.228438 6727 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 12:46:49.228539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:49.228519 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 12:46:49.725202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:49.725001 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:49.903502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:49.896963 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:50.226822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:50.226795 6727 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 12:46:50.227027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:50.227007 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 12:46:50.440824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:50.439993 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:46:50.440824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:50.440780 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:46:50.932790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:50.932742 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:51.015416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.015377 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9"] Oct 31 12:46:51.022102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.022079 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:51.029605 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.029449 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9"] Oct 31 12:46:51.219604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.217807 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759rj\" (UniqueName: \"kubernetes.io/projected/e21586c5-6d0e-4ded-9809-270fd1022e95-kube-api-access-759rj\") pod \"payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:51.219604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.217964 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21586c5-6d0e-4ded-9809-270fd1022e95-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:51.219604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.218584 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5c5815-dockercfg-ht77f\"" Oct 31 12:46:51.219604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.218664 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5c5815\"" Oct 31 12:46:51.241494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.228800 6727 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 12:46:51.241494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.228852 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 12:46:51.286486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.284024 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:51.415341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.374702 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rj\" (UniqueName: \"kubernetes.io/projected/e21586c5-6d0e-4ded-9809-270fd1022e95-kube-api-access-759rj\") pod \"payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:51.415341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.374864 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21586c5-6d0e-4ded-9809-270fd1022e95-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:51.416436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.416290 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21586c5-6d0e-4ded-9809-270fd1022e95-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:51.445090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.445058 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-759rj\" (UniqueName: \"kubernetes.io/projected/e21586c5-6d0e-4ded-9809-270fd1022e95-kube-api-access-759rj\") pod \"payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:51.641406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:51.641380 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:52.199527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.197112 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9"] Oct 31 12:46:52.205514 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:46:52.204207 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21586c5_6d0e_4ded_9809_270fd1022e95.slice/crio-cafc654e8465e0ff8f541c9712090277d066e49190f64eb97c16eedf2f7fe6d8 WatchSource:0}: Error finding container cafc654e8465e0ff8f541c9712090277d066e49190f64eb97c16eedf2f7fe6d8: Status 404 returned error can't find the container with id cafc654e8465e0ff8f541c9712090277d066e49190f64eb97c16eedf2f7fe6d8 Oct 31 12:46:52.228025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.227998 6727 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 12:46:52.228115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.228053 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 12:46:52.305047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.305020 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:52.635408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.635354 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" event={"ID":"e21586c5-6d0e-4ded-9809-270fd1022e95","Type":"ContainerStarted","Data":"fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113"} Oct 31 12:46:52.635655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.635439 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" event={"ID":"e21586c5-6d0e-4ded-9809-270fd1022e95","Type":"ContainerStarted","Data":"cafc654e8465e0ff8f541c9712090277d066e49190f64eb97c16eedf2f7fe6d8"} Oct 31 12:46:52.636712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.636091 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:46:52.690584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.682398 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" podStartSLOduration=2.682379468 podStartE2EDuration="2.682379468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:52.680665333 +0000 UTC m=+1123.495679726" watchObservedRunningTime="2026-10-31 12:46:52.682379468 +0000 UTC m=+1123.497393860" Oct 31 12:46:52.758558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:52.758520 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:53.096375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:53.096336 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:46:53.103696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:53.103666 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:53.229597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:53.226965 6727 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 12:46:53.229597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:53.227020 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 12:46:53.382558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:53.374129 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:53.815431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:53.815360 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:53.978493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:53.971144 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:54.226862 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:54.226822 6727 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 12:46:54.227071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:54.226890 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 12:46:54.550175 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:54.549993 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:55.231874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:55.231846 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:55.232345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:55.232322 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:55.233062 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:55.233035 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:46:55.355750 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:55.355721 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:56.284144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:56.284105 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:57.119125 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:57.119089 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:57.302283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:57.302238 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:57.453664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:57.453569 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:58.052782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.052742 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:58.575647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.575614 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:58.675353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.675323 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk"] Oct 31 12:46:58.680412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.680393 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:58.703846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.703804 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fb2b1d\"" Oct 31 12:46:58.713939 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.713911 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk"] Oct 31 12:46:58.714214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.714184 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f3751c-f307-4903-ab02-f236dc860401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:58.714573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.714541 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdqj\" (UniqueName: \"kubernetes.io/projected/46f3751c-f307-4903-ab02-f236dc860401-kube-api-access-fxdqj\") pod \"payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:58.794753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.794707 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:58.817429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.817394 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdqj\" (UniqueName: \"kubernetes.io/projected/46f3751c-f307-4903-ab02-f236dc860401-kube-api-access-fxdqj\") pod \"payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:58.817752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.817732 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f3751c-f307-4903-ab02-f236dc860401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:58.819128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.819103 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f3751c-f307-4903-ab02-f236dc860401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:58.834974 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.834908 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdqj\" (UniqueName: \"kubernetes.io/projected/46f3751c-f307-4903-ab02-f236dc860401-kube-api-access-fxdqj\") pod \"payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:58.970486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:58.969049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:59.007703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:59.007668 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:59.405676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:59.405646 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk"] Oct 31 12:46:59.405802 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:46:59.405685 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f3751c_f307_4903_ab02_f236dc860401.slice/crio-32a3e7b80c2fd326eba97ed7e8769c375cd4229216214450d2d286267563729f WatchSource:0}: Error finding container 32a3e7b80c2fd326eba97ed7e8769c375cd4229216214450d2d286267563729f: Status 404 returned error can't find the container with id 32a3e7b80c2fd326eba97ed7e8769c375cd4229216214450d2d286267563729f Oct 31 12:46:59.738623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:59.738535 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" event={"ID":"46f3751c-f307-4903-ab02-f236dc860401","Type":"ContainerStarted","Data":"a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a"} Oct 31 12:46:59.738623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:59.738598 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" event={"ID":"46f3751c-f307-4903-ab02-f236dc860401","Type":"ContainerStarted","Data":"32a3e7b80c2fd326eba97ed7e8769c375cd4229216214450d2d286267563729f"} Oct 31 12:46:59.739124 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:59.738719 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:46:59.757902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:59.757858 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" podStartSLOduration=1.757845715 podStartE2EDuration="1.757845715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:59.755120941 +0000 UTC m=+1130.570135332" watchObservedRunningTime="2026-10-31 12:46:59.757845715 +0000 UTC m=+1130.572860105" Oct 31 12:46:59.800909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:46:59.800888 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:00.343734 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:00.343697 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:47:00.517736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:00.517710 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:01.342144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:01.342110 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:02.034259 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:02.034227 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:02.722637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:02.722604 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:03.415900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:03.415865 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:03.648658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:03.648633 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:47:04.061599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.043515 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7"] Oct 31 12:47:04.101185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.101153 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7"] Oct 31 12:47:04.101548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.101521 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.119482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.106531 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5c5815-serving-cert\"" Oct 31 12:47:04.262993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.262941 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.263250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.263219 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-metrics-certs\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.263585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.263555 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.263852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.263735 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4zd\" (UniqueName: \"kubernetes.io/projected/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-kube-api-access-8s4zd\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.329154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.329117 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:04.366403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.366359 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4zd\" (UniqueName: \"kubernetes.io/projected/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-kube-api-access-8s4zd\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.366652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.366609 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.366829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.366807 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-metrics-certs\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.366946 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.366927 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.367999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.367972 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.368385 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:04.368364 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-serving-cert: secret "maas-api-e2e-api-a-5c5815-serving-cert" not found Oct 31 12:47:04.368497 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:04.368427 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls podName:433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c nodeName:}" failed. No retries permitted until 2026-10-31 12:47:04.868409692 +0000 UTC m=+1135.683424069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls") pod "maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" (UID: "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c") : secret "maas-api-e2e-api-a-5c5815-serving-cert" not found Oct 31 12:47:04.371594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.371565 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-metrics-certs\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.385169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.385141 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4zd\" (UniqueName: \"kubernetes.io/projected/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-kube-api-access-8s4zd\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.505366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.505320 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:04.678409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.678329 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:04.879898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.879858 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:04.882525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:04.882495 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls\") pod \"maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:05.133884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.133841 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:05.226184 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.226152 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:05.464964 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.464936 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7"] Oct 31 12:47:05.465627 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:47:05.465601 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433dd5d2_0296_45fb_bd5e_8b4b9b0bc25c.slice/crio-4630957d612eb9bdddbc4a7ca98edd72e8452da1dbc41fee269acde359afb36c WatchSource:0}: Error finding container 4630957d612eb9bdddbc4a7ca98edd72e8452da1dbc41fee269acde359afb36c: Status 404 returned error can't find the container with id 4630957d612eb9bdddbc4a7ca98edd72e8452da1dbc41fee269acde359afb36c Oct 31 12:47:05.603057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.603020 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf"] Oct 31 12:47:05.613968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.613910 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh"] Oct 31 12:47:05.709821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.709793 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:05.829531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.829497 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" event={"ID":"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c","Type":"ContainerStarted","Data":"4630957d612eb9bdddbc4a7ca98edd72e8452da1dbc41fee269acde359afb36c"} Oct 31 12:47:05.842424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:05.842388 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:06.038500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:06.032168 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:06.592181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:06.592150 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:06.840780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:06.840745 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" event={"ID":"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c","Type":"ContainerStarted","Data":"4ca962b657715081a3960bb12529287a4fe54435799507512f896ab53bc6c755"} Oct 31 12:47:06.840972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:06.840833 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:06.845255 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:06.845198 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:06.882372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:06.879485 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" podStartSLOduration=2.26833976 podStartE2EDuration="2.879447365s" totalImagesPullingTime="611.107605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:06.871497287 +0000 UTC m=+1137.686511691" watchObservedRunningTime="2026-10-31 12:47:06.879447365 +0000 UTC m=+1137.694461769" Oct 31 12:47:07.312877 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:07.312801 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:07.552528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:07.552491 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:08.508804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:08.508769 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:09.349802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:09.349773 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:10.112526 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.112453 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:10.604070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.604032 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" podUID="302a3e61-bc0d-485f-8a31-36b4111b2d36" containerName="payload-processing" containerID="cri-o://892d9ad6f252ee9165a95eb54b87f9326a402f4c51f0d1dbff80f0f2ef5dbc6b" gracePeriod=25 Oct 31 12:47:10.614226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.614199 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" podUID="66786b56-88d9-4568-b78e-0835cb1d1fab" containerName="payload-pre-processing" containerID="cri-o://2694633dc55f1e8724c24cef352a38249af589567703a409ba57292acb958006" gracePeriod=25 Oct 31 12:47:10.750584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.750109 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:47:10.811050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.811013 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:10.884696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.884660 6727 generic.go:356] "Generic (PLEG): container finished" podID="302a3e61-bc0d-485f-8a31-36b4111b2d36" containerID="892d9ad6f252ee9165a95eb54b87f9326a402f4c51f0d1dbff80f0f2ef5dbc6b" exitCode=0 Oct 31 12:47:10.884844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.884762 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" event={"ID":"302a3e61-bc0d-485f-8a31-36b4111b2d36","Type":"ContainerDied","Data":"892d9ad6f252ee9165a95eb54b87f9326a402f4c51f0d1dbff80f0f2ef5dbc6b"} Oct 31 12:47:10.887572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.887153 6727 generic.go:356] "Generic (PLEG): container finished" podID="66786b56-88d9-4568-b78e-0835cb1d1fab" containerID="2694633dc55f1e8724c24cef352a38249af589567703a409ba57292acb958006" exitCode=0 Oct 31 12:47:10.887572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.887426 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" event={"ID":"66786b56-88d9-4568-b78e-0835cb1d1fab","Type":"ContainerDied","Data":"2694633dc55f1e8724c24cef352a38249af589567703a409ba57292acb958006"} Oct 31 12:47:10.940434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:10.940344 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:11.093635 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.093610 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:47:11.096645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.096623 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:47:11.150043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.149958 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:11.249407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.249379 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302a3e61-bc0d-485f-8a31-36b4111b2d36-kube-api-access-gj4nx\" (UniqueName: \"kubernetes.io/projected/302a3e61-bc0d-485f-8a31-36b4111b2d36-kube-api-access-gj4nx\") pod \"302a3e61-bc0d-485f-8a31-36b4111b2d36\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " Oct 31 12:47:11.249574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.249510 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302a3e61-bc0d-485f-8a31-36b4111b2d36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a3e61-bc0d-485f-8a31-36b4111b2d36-plugins-config-volume\") pod \"302a3e61-bc0d-485f-8a31-36b4111b2d36\" (UID: \"302a3e61-bc0d-485f-8a31-36b4111b2d36\") " Oct 31 12:47:11.249574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.249543 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66786b56-88d9-4568-b78e-0835cb1d1fab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66786b56-88d9-4568-b78e-0835cb1d1fab-plugins-config-volume\") pod \"66786b56-88d9-4568-b78e-0835cb1d1fab\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " Oct 31 12:47:11.249726 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.249586 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66786b56-88d9-4568-b78e-0835cb1d1fab-kube-api-access-nqvx9\" (UniqueName: \"kubernetes.io/projected/66786b56-88d9-4568-b78e-0835cb1d1fab-kube-api-access-nqvx9\") pod \"66786b56-88d9-4568-b78e-0835cb1d1fab\" (UID: \"66786b56-88d9-4568-b78e-0835cb1d1fab\") " Oct 31 12:47:11.249945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.249907 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302a3e61-bc0d-485f-8a31-36b4111b2d36-plugins-config-volume" pod "302a3e61-bc0d-485f-8a31-36b4111b2d36" (UID: "302a3e61-bc0d-485f-8a31-36b4111b2d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:47:11.249945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.249909 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66786b56-88d9-4568-b78e-0835cb1d1fab-plugins-config-volume" pod "66786b56-88d9-4568-b78e-0835cb1d1fab" (UID: "66786b56-88d9-4568-b78e-0835cb1d1fab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:47:11.250205 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.250185 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a3e61-bc0d-485f-8a31-36b4111b2d36-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:11.250264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.250211 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66786b56-88d9-4568-b78e-0835cb1d1fab-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:11.251621 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.251603 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302a3e61-bc0d-485f-8a31-36b4111b2d36-kube-api-access-gj4nx" pod "302a3e61-bc0d-485f-8a31-36b4111b2d36" (UID: "302a3e61-bc0d-485f-8a31-36b4111b2d36"). InnerVolumeSpecName "kube-api-access-gj4nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:47:11.251664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.251652 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66786b56-88d9-4568-b78e-0835cb1d1fab-kube-api-access-nqvx9" pod "66786b56-88d9-4568-b78e-0835cb1d1fab" (UID: "66786b56-88d9-4568-b78e-0835cb1d1fab"). InnerVolumeSpecName "kube-api-access-nqvx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:47:11.352565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.352534 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj4nx\" (UniqueName: \"kubernetes.io/projected/302a3e61-bc0d-485f-8a31-36b4111b2d36-kube-api-access-gj4nx\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:11.352565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.352565 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqvx9\" (UniqueName: \"kubernetes.io/projected/66786b56-88d9-4568-b78e-0835cb1d1fab-kube-api-access-nqvx9\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:11.807185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.807151 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:11.900241 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.900204 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" event={"ID":"302a3e61-bc0d-485f-8a31-36b4111b2d36","Type":"ContainerDied","Data":"b6f08b1dc98c39c69b9447d4c4ad2c0befbf9cf50ac2704519970415d02cf388"} Oct 31 12:47:11.900432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.900268 6727 scope.go:122] "RemoveContainer" containerID="892d9ad6f252ee9165a95eb54b87f9326a402f4c51f0d1dbff80f0f2ef5dbc6b" Oct 31 12:47:11.900432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.900211 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf" Oct 31 12:47:11.901922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.901899 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" Oct 31 12:47:11.902064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.901925 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh" event={"ID":"66786b56-88d9-4568-b78e-0835cb1d1fab","Type":"ContainerDied","Data":"4fffea1118987128a0d7512f94f8954cd417556795ad608063353ef8bdbac7af"} Oct 31 12:47:11.934061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.933757 6727 scope.go:122] "RemoveContainer" containerID="2694633dc55f1e8724c24cef352a38249af589567703a409ba57292acb958006" Oct 31 12:47:11.942196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.941919 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh"] Oct 31 12:47:11.949580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.948493 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d310d-5dd558cc67-hqsqh"] Oct 31 12:47:11.959826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.959792 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf"] Oct 31 12:47:11.964794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:11.964766 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d310d-7889d55669-smsmf"] Oct 31 12:47:12.643862 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:12.643787 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:12.854565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:12.854535 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:47:13.303666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:13.303632 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302a3e61-bc0d-485f-8a31-36b4111b2d36" path="/var/lib/kubelet/pods/302a3e61-bc0d-485f-8a31-36b4111b2d36/volumes" Oct 31 12:47:13.304207 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:13.304184 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66786b56-88d9-4568-b78e-0835cb1d1fab" path="/var/lib/kubelet/pods/66786b56-88d9-4568-b78e-0835cb1d1fab/volumes" Oct 31 12:47:13.505019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:13.504000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:14.212411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:14.212377 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:14.937576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:14.935910 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:15.631747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:15.631709 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:16.390043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:16.390008 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:17.217806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:17.217770 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:17.785295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:17.785256 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:17.929440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:17.929402 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:18.096090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:18.096056 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:18.258895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:18.258864 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:18.832642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:18.832606 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:19.982403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:19.982367 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:20.972619 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:20.972577 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:21.675245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:21.675217 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:22.354875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:22.354839 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:23.077877 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:23.077844 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:23.848641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:23.848604 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:23.983712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:23.983676 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:24.494020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:24.493965 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:25.032938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:25.032907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:25.189966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:25.189904 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:25.661900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:25.661845 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:26.300946 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:26.299647 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:26.470540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:26.470500 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:27.196930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:27.196897 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:27.334271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:27.334242 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:27.514207 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:27.514125 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:27.961179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:27.961145 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:28.100656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.100621 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:28.344849 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.344815 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq"] Oct 31 12:47:28.345960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.345939 6727 cpu_manager.go:414] "containerMap: removing container" podUID="302a3e61-bc0d-485f-8a31-36b4111b2d36" containerName="payload-processing" Oct 31 12:47:28.345960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.345958 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302a3e61-bc0d-485f-8a31-36b4111b2d36" containerName="payload-processing" Oct 31 12:47:28.345960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.345969 6727 cpu_manager.go:414] "containerMap: removing container" podUID="66786b56-88d9-4568-b78e-0835cb1d1fab" containerName="payload-pre-processing" Oct 31 12:47:28.345960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.345975 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66786b56-88d9-4568-b78e-0835cb1d1fab" containerName="payload-pre-processing" Oct 31 12:47:28.346264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.346131 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="302a3e61-bc0d-485f-8a31-36b4111b2d36" containerName="payload-processing" Oct 31 12:47:28.346264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.346140 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="66786b56-88d9-4568-b78e-0835cb1d1fab" containerName="payload-pre-processing" Oct 31 12:47:28.350128 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.350110 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.362992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.362285 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d4dd68-openshift-default-dockercfg-rqjmm\"" Oct 31 12:47:28.368804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.368785 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq"] Oct 31 12:47:28.441818 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.441794 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:28.470824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.470791 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-certs\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.470958 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.470839 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cq88\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-kube-api-access-5cq88\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.470958 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.470937 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-envoy\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.471076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.471059 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f585440-9878-4540-b806-6704375dc496-istiod-ca-cert\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.471196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.471173 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-istio-token\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.471250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.471230 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f585440-9878-4540-b806-6704375dc496-istio-podinfo\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.471335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.471317 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-credential-socket\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.471412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.471399 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-socket\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.471505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.471488 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-data\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.582551 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-socket\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.582655 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-data\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.582853 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-certs\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.582912 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cq88\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-kube-api-access-5cq88\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.582976 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-envoy\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.583124 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f585440-9878-4540-b806-6704375dc496-istiod-ca-cert\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.584525 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-certs\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.584782 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-socket\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.588490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.585005 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-data\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.597525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.594561 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-istio-token\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.597525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.594670 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f585440-9878-4540-b806-6704375dc496-istio-podinfo\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.597525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.594721 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-credential-socket\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.597525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.595152 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-credential-socket\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.606492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.602534 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f585440-9878-4540-b806-6704375dc496-istiod-ca-cert\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.646591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.644558 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f585440-9878-4540-b806-6704375dc496-istio-podinfo\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.661485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.646980 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-envoy\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.691484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.688622 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cq88\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-kube-api-access-5cq88\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.718162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.711676 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-istio-token\") pod \"e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:28.718162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.711988 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:28.970446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:28.970365 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:29.258624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.258554 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:29.275478 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:47:29.275436 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f585440_9878_4540_b806_6704375dc496.slice/crio-2b3b8bf9dbda217ced0d760802b6cb5d1bd615f41638af1614adb4065c8800fb WatchSource:0}: Error finding container 2b3b8bf9dbda217ced0d760802b6cb5d1bd615f41638af1614adb4065c8800fb: Status 404 returned error can't find the container with id 2b3b8bf9dbda217ced0d760802b6cb5d1bd615f41638af1614adb4065c8800fb Oct 31 12:47:29.275590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.275506 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq"] Oct 31 12:47:29.278285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.278250 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:47:29.278393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.278317 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:47:29.278393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.278351 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:47:29.399182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.399135 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:29.796308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.796257 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:29.989494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:29.988984 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:30.128090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:30.128057 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" event={"ID":"7f585440-9878-4540-b806-6704375dc496","Type":"ContainerStarted","Data":"58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120"} Oct 31 12:47:30.128271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:30.128119 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" event={"ID":"7f585440-9878-4540-b806-6704375dc496","Type":"ContainerStarted","Data":"2b3b8bf9dbda217ced0d760802b6cb5d1bd615f41638af1614adb4065c8800fb"} Oct 31 12:47:30.152038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:30.151670 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podStartSLOduration=2.151651508 podStartE2EDuration="2.151651508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:30.150957999 +0000 UTC m=+1160.965972392" watchObservedRunningTime="2026-10-31 12:47:30.151651508 +0000 UTC m=+1160.966665902" Oct 31 12:47:30.353381 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:30.353339 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:47:30.468420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:30.468316 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:30.861498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:30.861458 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:30.970580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:30.970550 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:30.976327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:30.976305 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:31.057650 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:31.057593 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:31.135575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:31.135501 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:31.136341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:31.136323 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:47:31.543659 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:31.543553 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:32.104602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.104505 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:32.180947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.180915 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9"] Oct 31 12:47:32.186930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.186594 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.208158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.208134 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9"] Oct 31 12:47:32.291932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.276016 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:32.321789 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.290693 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-787316\"" Oct 31 12:47:32.322090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.290843 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-787316-dockercfg-s8bkn\"" Oct 31 12:47:32.332651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.332585 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4858fc3-28c5-468e-95ec-04009bcb9b32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.333293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.333259 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzp2\" (UniqueName: \"kubernetes.io/projected/c4858fc3-28c5-468e-95ec-04009bcb9b32-kube-api-access-rlzp2\") pod \"payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.477123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.477031 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4858fc3-28c5-468e-95ec-04009bcb9b32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.477834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.477801 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzp2\" (UniqueName: \"kubernetes.io/projected/c4858fc3-28c5-468e-95ec-04009bcb9b32-kube-api-access-rlzp2\") pod \"payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.479582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.479559 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4858fc3-28c5-468e-95ec-04009bcb9b32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.533485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.533081 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzp2\" (UniqueName: \"kubernetes.io/projected/c4858fc3-28c5-468e-95ec-04009bcb9b32-kube-api-access-rlzp2\") pod \"payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.720776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.720745 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:32.818612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.818528 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:32.933072 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:32.933027 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:33.086486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:33.077550 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:33.181497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:33.181425 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9"] Oct 31 12:47:33.182524 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:47:33.182452 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4858fc3_28c5_468e_95ec_04009bcb9b32.slice/crio-952b9af884a81675cbae74cbd7f8222d539dd90c4370bbedbcb684edc9113c0c WatchSource:0}: Error finding container 952b9af884a81675cbae74cbd7f8222d539dd90c4370bbedbcb684edc9113c0c: Status 404 returned error can't find the container with id 952b9af884a81675cbae74cbd7f8222d539dd90c4370bbedbcb684edc9113c0c Oct 31 12:47:33.918900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:33.918849 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:34.074485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.074079 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:34.243546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.242161 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" event={"ID":"c4858fc3-28c5-468e-95ec-04009bcb9b32","Type":"ContainerStarted","Data":"ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990"} Oct 31 12:47:34.243546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.242216 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" event={"ID":"c4858fc3-28c5-468e-95ec-04009bcb9b32","Type":"ContainerStarted","Data":"952b9af884a81675cbae74cbd7f8222d539dd90c4370bbedbcb684edc9113c0c"} Oct 31 12:47:34.243546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.242417 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:34.281696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.280919 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:34.293995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.293749 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" podStartSLOduration=2.293730989 podStartE2EDuration="2.293730989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:34.283327243 +0000 UTC m=+1165.098341635" watchObservedRunningTime="2026-10-31 12:47:34.293730989 +0000 UTC m=+1165.108745380" Oct 31 12:47:34.866557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.866520 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5"] Oct 31 12:47:34.866831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.866795 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" podUID="81df80ca-28f7-465a-a4c6-09bbbea0b503" containerName="maas-api" containerID="cri-o://e11655b442aff96db2acadd0c681ed6772fe4b5b2d166832aa11cdb6164c892d" gracePeriod=30 Oct 31 12:47:34.972150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:34.972107 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:35.009042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.009004 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2"] Oct 31 12:47:35.017638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.017608 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2"] Oct 31 12:47:35.017773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.017712 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.054703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.053980 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8w7n\" (UniqueName: \"kubernetes.io/projected/93c56ad8-7acd-496c-9b0d-c9569405e026-kube-api-access-m8w7n\") pod \"payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.054703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.054419 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c56ad8-7acd-496c-9b0d-c9569405e026-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.096403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.096377 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d4dd68\"" Oct 31 12:47:35.117780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.117715 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:35.161693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.159828 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8w7n\" (UniqueName: \"kubernetes.io/projected/93c56ad8-7acd-496c-9b0d-c9569405e026-kube-api-access-m8w7n\") pod \"payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.161693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.160278 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c56ad8-7acd-496c-9b0d-c9569405e026-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.161693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.161294 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c56ad8-7acd-496c-9b0d-c9569405e026-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.250489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.240504 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8w7n\" (UniqueName: \"kubernetes.io/projected/93c56ad8-7acd-496c-9b0d-c9569405e026-kube-api-access-m8w7n\") pod \"payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.360630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.359526 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:35.396180 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.395981 6727 generic.go:356] "Generic (PLEG): container finished" podID="81df80ca-28f7-465a-a4c6-09bbbea0b503" containerID="e11655b442aff96db2acadd0c681ed6772fe4b5b2d166832aa11cdb6164c892d" exitCode=0 Oct 31 12:47:35.402678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.402642 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" event={"ID":"81df80ca-28f7-465a-a4c6-09bbbea0b503","Type":"ContainerDied","Data":"e11655b442aff96db2acadd0c681ed6772fe4b5b2d166832aa11cdb6164c892d"} Oct 31 12:47:35.517332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.509874 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:35.517332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.513616 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:47:35.582489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.578985 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81df80ca-28f7-465a-a4c6-09bbbea0b503-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81df80ca-28f7-465a-a4c6-09bbbea0b503-openshift-service-ca\") pod \"81df80ca-28f7-465a-a4c6-09bbbea0b503\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " Oct 31 12:47:35.582489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.579382 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81df80ca-28f7-465a-a4c6-09bbbea0b503-openshift-service-ca" pod "81df80ca-28f7-465a-a4c6-09bbbea0b503" (UID: "81df80ca-28f7-465a-a4c6-09bbbea0b503"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:47:35.582489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.579725 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-maas-api-tls\") pod \"81df80ca-28f7-465a-a4c6-09bbbea0b503\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " Oct 31 12:47:35.582489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.580142 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-metrics-certs\") pod \"81df80ca-28f7-465a-a4c6-09bbbea0b503\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " Oct 31 12:47:35.582489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.580209 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81df80ca-28f7-465a-a4c6-09bbbea0b503-kube-api-access-4fc9h\" (UniqueName: \"kubernetes.io/projected/81df80ca-28f7-465a-a4c6-09bbbea0b503-kube-api-access-4fc9h\") pod \"81df80ca-28f7-465a-a4c6-09bbbea0b503\" (UID: \"81df80ca-28f7-465a-a4c6-09bbbea0b503\") " Oct 31 12:47:35.582489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.581141 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81df80ca-28f7-465a-a4c6-09bbbea0b503-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:35.596517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.593035 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81df80ca-28f7-465a-a4c6-09bbbea0b503-kube-api-access-4fc9h" pod "81df80ca-28f7-465a-a4c6-09bbbea0b503" (UID: "81df80ca-28f7-465a-a4c6-09bbbea0b503"). InnerVolumeSpecName "kube-api-access-4fc9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:47:35.596517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.593143 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-metrics-certs" pod "81df80ca-28f7-465a-a4c6-09bbbea0b503" (UID: "81df80ca-28f7-465a-a4c6-09bbbea0b503"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:47:35.596517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.593232 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-maas-api-tls" pod "81df80ca-28f7-465a-a4c6-09bbbea0b503" (UID: "81df80ca-28f7-465a-a4c6-09bbbea0b503"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:47:35.679536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.668774 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:35.685736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.684079 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:35.685736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.684114 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fc9h\" (UniqueName: \"kubernetes.io/projected/81df80ca-28f7-465a-a4c6-09bbbea0b503-kube-api-access-4fc9h\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:35.685736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.684129 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81df80ca-28f7-465a-a4c6-09bbbea0b503-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:35.837294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:35.832894 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2"] Oct 31 12:47:36.061367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.061344 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:36.214637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.214611 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:36.251574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.251489 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv"] Oct 31 12:47:36.252435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.252411 6727 cpu_manager.go:414] "containerMap: removing container" podUID="81df80ca-28f7-465a-a4c6-09bbbea0b503" containerName="maas-api" Oct 31 12:47:36.252435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.252434 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81df80ca-28f7-465a-a4c6-09bbbea0b503" containerName="maas-api" Oct 31 12:47:36.252731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.252715 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="81df80ca-28f7-465a-a4c6-09bbbea0b503" containerName="maas-api" Oct 31 12:47:36.256639 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.256596 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.269684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.269302 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv"] Oct 31 12:47:36.270557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.270072 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-015701\"" Oct 31 12:47:36.297312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.297278 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15eda72c-a4ed-408d-97d0-006037e7bd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.298396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.297613 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtsnc\" (UniqueName: \"kubernetes.io/projected/15eda72c-a4ed-408d-97d0-006037e7bd5f-kube-api-access-xtsnc\") pod \"payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.325505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.325428 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-015701-dockercfg-hz68c\"" Oct 31 12:47:36.400489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.397877 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:36.400489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.399288 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtsnc\" (UniqueName: \"kubernetes.io/projected/15eda72c-a4ed-408d-97d0-006037e7bd5f-kube-api-access-xtsnc\") pod \"payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.410525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.405731 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15eda72c-a4ed-408d-97d0-006037e7bd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.410525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.407080 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15eda72c-a4ed-408d-97d0-006037e7bd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.560797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.560621 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtsnc\" (UniqueName: \"kubernetes.io/projected/15eda72c-a4ed-408d-97d0-006037e7bd5f-kube-api-access-xtsnc\") pod \"payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.562078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.561900 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" event={"ID":"93c56ad8-7acd-496c-9b0d-c9569405e026","Type":"ContainerStarted","Data":"26bf36c22b93e0b94dab1ca55482dbfe10b2e062a890cf7b54cdc72b877eecc4"} Oct 31 12:47:36.562078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.561968 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" event={"ID":"93c56ad8-7acd-496c-9b0d-c9569405e026","Type":"ContainerStarted","Data":"dfaf9dd5450caba80741265326464ed2497c46f7123d0f24472c2fe991d8e7ee"} Oct 31 12:47:36.563533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.563451 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:36.564975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.564954 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" event={"ID":"81df80ca-28f7-465a-a4c6-09bbbea0b503","Type":"ContainerDied","Data":"f1cf983d3ae02a6b9a7b9f0c88ebde9c7ad79369e9a315f7e5c2f89ef4f8cb9f"} Oct 31 12:47:36.565165 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.565135 6727 scope.go:122] "RemoveContainer" containerID="e11655b442aff96db2acadd0c681ed6772fe4b5b2d166832aa11cdb6164c892d" Oct 31 12:47:36.565275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.564971 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5" Oct 31 12:47:36.582506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.581967 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:36.606539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.604501 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" podStartSLOduration=2.604480664 podStartE2EDuration="2.604480664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:36.596280474 +0000 UTC m=+1167.411294873" watchObservedRunningTime="2026-10-31 12:47:36.604480664 +0000 UTC m=+1167.419495051" Oct 31 12:47:36.750058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.748770 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5"] Oct 31 12:47:36.755692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.755658 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d310d-d796f97d8-hpfk5"] Oct 31 12:47:36.867971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:36.867937 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:37.009054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:37.009016 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv"] Oct 31 12:47:37.009788 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:47:37.009749 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15eda72c_a4ed_408d_97d0_006037e7bd5f.slice/crio-4d6cc09e01abe3d9fff43542574271d4603e891435f32506c15d74a0287f21d9 WatchSource:0}: Error finding container 4d6cc09e01abe3d9fff43542574271d4603e891435f32506c15d74a0287f21d9: Status 404 returned error can't find the container with id 4d6cc09e01abe3d9fff43542574271d4603e891435f32506c15d74a0287f21d9 Oct 31 12:47:37.304857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:37.304828 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81df80ca-28f7-465a-a4c6-09bbbea0b503" path="/var/lib/kubelet/pods/81df80ca-28f7-465a-a4c6-09bbbea0b503/volumes" Oct 31 12:47:37.631483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:37.620449 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" event={"ID":"15eda72c-a4ed-408d-97d0-006037e7bd5f","Type":"ContainerStarted","Data":"2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429"} Oct 31 12:47:37.631483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:37.629581 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" event={"ID":"15eda72c-a4ed-408d-97d0-006037e7bd5f","Type":"ContainerStarted","Data":"4d6cc09e01abe3d9fff43542574271d4603e891435f32506c15d74a0287f21d9"} Oct 31 12:47:37.631483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:37.629624 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:37.684487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:37.683748 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" podStartSLOduration=1.683729493 podStartE2EDuration="1.683729493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:37.682646771 +0000 UTC m=+1168.497661164" watchObservedRunningTime="2026-10-31 12:47:37.683729493 +0000 UTC m=+1168.498743886" Oct 31 12:47:37.975304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:37.975206 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:38.793505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:38.792919 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:39.749829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:39.749794 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:40.691935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:40.691907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:41.441850 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:41.441767 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:42.120972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:42.120924 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:42.920853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:42.920815 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:43.299495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:43.299397 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:43.489660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:43.489621 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:43.675388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:43.675361 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:43.828414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:43.828382 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:44.450247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:44.450213 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:44.782858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:44.782582 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:44.924793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:44.924748 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:45.069850 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:45.069818 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:45.401028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:45.400948 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:47:45.407723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:45.407692 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:45.616261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:45.616230 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm"] Oct 31 12:47:45.617023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:45.616993 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" podUID="feb660f2-f9ad-4e11-86af-51690fc91f24" containerName="maas-api" containerID="cri-o://6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa" gracePeriod=30 Oct 31 12:47:45.693516 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:45.693394 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:46.024718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:45.990000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:46.142357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.142321 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:46.205965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.205934 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:47:46.288675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.288599 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:46.339559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.339523 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls\") pod \"feb660f2-f9ad-4e11-86af-51690fc91f24\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " Oct 31 12:47:46.339765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.339572 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb660f2-f9ad-4e11-86af-51690fc91f24-kube-api-access-hxgnm\" (UniqueName: \"kubernetes.io/projected/feb660f2-f9ad-4e11-86af-51690fc91f24-kube-api-access-hxgnm\") pod \"feb660f2-f9ad-4e11-86af-51690fc91f24\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " Oct 31 12:47:46.339959 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.339922 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb660f2-f9ad-4e11-86af-51690fc91f24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb660f2-f9ad-4e11-86af-51690fc91f24-openshift-service-ca\") pod \"feb660f2-f9ad-4e11-86af-51690fc91f24\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " Oct 31 12:47:46.340149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.340065 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-metrics-certs\") pod \"feb660f2-f9ad-4e11-86af-51690fc91f24\" (UID: \"feb660f2-f9ad-4e11-86af-51690fc91f24\") " Oct 31 12:47:46.342902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.340367 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb660f2-f9ad-4e11-86af-51690fc91f24-openshift-service-ca" pod "feb660f2-f9ad-4e11-86af-51690fc91f24" (UID: "feb660f2-f9ad-4e11-86af-51690fc91f24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:47:46.342902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.341642 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb660f2-f9ad-4e11-86af-51690fc91f24-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:46.342902 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.342379 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls" pod "feb660f2-f9ad-4e11-86af-51690fc91f24" (UID: "feb660f2-f9ad-4e11-86af-51690fc91f24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:47:46.347849 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.346951 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb660f2-f9ad-4e11-86af-51690fc91f24-kube-api-access-hxgnm" pod "feb660f2-f9ad-4e11-86af-51690fc91f24" (UID: "feb660f2-f9ad-4e11-86af-51690fc91f24"). InnerVolumeSpecName "kube-api-access-hxgnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:47:46.347849 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.346963 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-metrics-certs" pod "feb660f2-f9ad-4e11-86af-51690fc91f24" (UID: "feb660f2-f9ad-4e11-86af-51690fc91f24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:47:46.443572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.443492 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:46.443572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.443532 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb660f2-f9ad-4e11-86af-51690fc91f24-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:46.443572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.443548 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxgnm\" (UniqueName: \"kubernetes.io/projected/feb660f2-f9ad-4e11-86af-51690fc91f24-kube-api-access-hxgnm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:46.743448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.743417 6727 generic.go:356] "Generic (PLEG): container finished" podID="feb660f2-f9ad-4e11-86af-51690fc91f24" containerID="6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa" exitCode=0 Oct 31 12:47:46.743906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.743509 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" Oct 31 12:47:46.743906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.743526 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" event={"ID":"feb660f2-f9ad-4e11-86af-51690fc91f24","Type":"ContainerDied","Data":"6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa"} Oct 31 12:47:46.743906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.743587 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm" event={"ID":"feb660f2-f9ad-4e11-86af-51690fc91f24","Type":"ContainerDied","Data":"409ec1b31ebce3d64d4e32da40549cf8bf917c30c075177e0e8026ba6eb3b02b"} Oct 31 12:47:46.743906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.743617 6727 scope.go:122] "RemoveContainer" containerID="6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa" Oct 31 12:47:46.757338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.756967 6727 scope.go:122] "RemoveContainer" containerID="6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa" Oct 31 12:47:46.757338 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:46.757255 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa\": container with ID starting with 6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa not found: ID does not exist" containerID="6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa" Oct 31 12:47:46.757338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.757281 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa"} err="failed to get container status \"6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa\": rpc error: code = NotFound desc = could not find container \"6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa\": container with ID starting with 6ab33d18bf12b2e89f2bd4fd8cf0e588a23e421d2d4f980edf061612dcaaa9fa not found: ID does not exist" Oct 31 12:47:46.769959 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.769936 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm"] Oct 31 12:47:46.774325 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:46.774304 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-bjlmm"] Oct 31 12:47:47.168952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.168920 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn"] Oct 31 12:47:47.169900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.169881 6727 cpu_manager.go:414] "containerMap: removing container" podUID="feb660f2-f9ad-4e11-86af-51690fc91f24" containerName="maas-api" Oct 31 12:47:47.169969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.169901 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb660f2-f9ad-4e11-86af-51690fc91f24" containerName="maas-api" Oct 31 12:47:47.170076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.170064 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb660f2-f9ad-4e11-86af-51690fc91f24" containerName="maas-api" Oct 31 12:47:47.175632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.175607 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.190515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.190457 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn"] Oct 31 12:47:47.204580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.204554 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-787316-serving-cert\"" Oct 31 12:47:47.255025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.254980 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-metrics-certs\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.255211 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.255050 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.255278 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.255210 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxd5\" (UniqueName: \"kubernetes.io/projected/4271c7a4-9c32-4b80-b9b0-9615ba75612a-kube-api-access-5dxd5\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.255333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.255304 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271c7a4-9c32-4b80-b9b0-9615ba75612a-openshift-service-ca\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.304756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.304722 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb660f2-f9ad-4e11-86af-51690fc91f24" path="/var/lib/kubelet/pods/feb660f2-f9ad-4e11-86af-51690fc91f24/volumes" Oct 31 12:47:47.357589 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.357539 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxd5\" (UniqueName: \"kubernetes.io/projected/4271c7a4-9c32-4b80-b9b0-9615ba75612a-kube-api-access-5dxd5\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.357794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.357688 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271c7a4-9c32-4b80-b9b0-9615ba75612a-openshift-service-ca\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.358117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.358047 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-metrics-certs\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.358117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.358107 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.358348 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:47.358308 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-787316-serving-cert: secret "maas-api-e2e-api-b-787316-serving-cert" not found Oct 31 12:47:47.358442 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:47.358374 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls podName:4271c7a4-9c32-4b80-b9b0-9615ba75612a nodeName:}" failed. No retries permitted until 2026-10-31 12:47:47.858354648 +0000 UTC m=+1178.673369024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls") pod "maas-api-e2e-api-b-787316-586f895476-dcqqn" (UID: "4271c7a4-9c32-4b80-b9b0-9615ba75612a") : secret "maas-api-e2e-api-b-787316-serving-cert" not found Oct 31 12:47:47.360404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.360377 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271c7a4-9c32-4b80-b9b0-9615ba75612a-openshift-service-ca\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.362251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.362223 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-metrics-certs\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.396543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.396508 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxd5\" (UniqueName: \"kubernetes.io/projected/4271c7a4-9c32-4b80-b9b0-9615ba75612a-kube-api-access-5dxd5\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.401012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.400985 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:47.595091 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.595062 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:47.623298 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.623273 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:47:47.722101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.721895 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:47.867833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.867761 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:47.870594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:47.870566 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls\") pod \"maas-api-e2e-api-b-787316-586f895476-dcqqn\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:48.096080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:48.096042 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:48.303990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:48.303954 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:48.402014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:48.401963 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn"] Oct 31 12:47:48.409763 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:47:48.409728 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4271c7a4_9c32_4b80_b9b0_9615ba75612a.slice/crio-ed1c83b81fdf39cf863e4df7f1887aa6ace2fd3742ed21bba656fcaf673f7c6f WatchSource:0}: Error finding container ed1c83b81fdf39cf863e4df7f1887aa6ace2fd3742ed21bba656fcaf673f7c6f: Status 404 returned error can't find the container with id ed1c83b81fdf39cf863e4df7f1887aa6ace2fd3742ed21bba656fcaf673f7c6f Oct 31 12:47:48.629281 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:48.629253 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:47:48.773390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:48.773301 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" event={"ID":"4271c7a4-9c32-4b80-b9b0-9615ba75612a","Type":"ContainerStarted","Data":"ed1c83b81fdf39cf863e4df7f1887aa6ace2fd3742ed21bba656fcaf673f7c6f"} Oct 31 12:47:49.251124 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.244347 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:49.786436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.786351 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" event={"ID":"4271c7a4-9c32-4b80-b9b0-9615ba75612a","Type":"ContainerStarted","Data":"a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984"} Oct 31 12:47:49.786436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.786427 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:49.809047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.809006 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" podStartSLOduration=2.367781484 podStartE2EDuration="2.808992471s" totalImagesPullingTime="441.210987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:49.805664184 +0000 UTC m=+1180.620678577" watchObservedRunningTime="2026-10-31 12:47:49.808992471 +0000 UTC m=+1180.624006883" Oct 31 12:47:49.868158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.868122 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl"] Oct 31 12:47:49.878214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.876599 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:49.883846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.883811 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl"] Oct 31 12:47:49.888527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.888458 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d4dd68-serving-cert\"" Oct 31 12:47:49.967908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:49.967875 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:50.012785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.012754 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64bea21e-91b0-4b3d-9a54-1f147f79e546-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.013061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.013035 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.013179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.013164 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q52fm\" (UniqueName: \"kubernetes.io/projected/64bea21e-91b0-4b3d-9a54-1f147f79e546-kube-api-access-q52fm\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.013358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.013338 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.114672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.114640 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.114960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.114704 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:50.114960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.114765 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52fm\" (UniqueName: \"kubernetes.io/projected/64bea21e-91b0-4b3d-9a54-1f147f79e546-kube-api-access-q52fm\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.114960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.114929 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.115173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.115024 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64bea21e-91b0-4b3d-9a54-1f147f79e546-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.115845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.115824 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64bea21e-91b0-4b3d-9a54-1f147f79e546-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.118158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.118134 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.118268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.118253 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.128865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.128836 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52fm\" (UniqueName: \"kubernetes.io/projected/64bea21e-91b0-4b3d-9a54-1f147f79e546-kube-api-access-q52fm\") pod \"maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.196178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.196143 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:50.339742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.339711 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:50.509940 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.497052 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl"] Oct 31 12:47:50.801079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.800968 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" event={"ID":"64bea21e-91b0-4b3d-9a54-1f147f79e546","Type":"ContainerStarted","Data":"833253b22b961e39f0214a5d0b6afa074c7bb07db7d484d560207bf0309aa451"} Oct 31 12:47:50.861110 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.861075 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns"] Oct 31 12:47:50.878655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.874535 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:50.888452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.887699 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-015701-serving-cert\"" Oct 31 12:47:50.903061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.903031 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns"] Oct 31 12:47:50.943228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:50.943198 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:51.040606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.040118 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5282b-eb99-4871-bc7f-44344133ff6d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.040606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.040253 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-metrics-certs\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.040606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.040330 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.044061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.043932 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k864\" (UniqueName: \"kubernetes.io/projected/afd5282b-eb99-4871-bc7f-44344133ff6d-kube-api-access-5k864\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.145889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.145853 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.146056 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:51.146018 6727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-serving-cert: secret "maas-api-e2e-inf-a-015701-serving-cert" not found Oct 31 12:47:51.146147 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:51.146082 6727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls podName:afd5282b-eb99-4871-bc7f-44344133ff6d nodeName:}" failed. No retries permitted until 2026-10-31 12:47:51.646067038 +0000 UTC m=+1182.461081407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls") pod "maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" (UID: "afd5282b-eb99-4871-bc7f-44344133ff6d") : secret "maas-api-e2e-inf-a-015701-serving-cert" not found Oct 31 12:47:51.146147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.146100 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k864\" (UniqueName: \"kubernetes.io/projected/afd5282b-eb99-4871-bc7f-44344133ff6d-kube-api-access-5k864\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.146243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.146232 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5282b-eb99-4871-bc7f-44344133ff6d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.146336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.146320 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-metrics-certs\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.146901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.146880 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5282b-eb99-4871-bc7f-44344133ff6d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.150156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.150134 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-metrics-certs\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.192686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.192656 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k864\" (UniqueName: \"kubernetes.io/projected/afd5282b-eb99-4871-bc7f-44344133ff6d-kube-api-access-5k864\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.279446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.279414 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:51.427688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.427617 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:51.656340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.656299 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.696535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.687177 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls\") pod \"maas-api-e2e-inf-a-015701-67cc6556dd-n6cns\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.812150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.812110 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" event={"ID":"64bea21e-91b0-4b3d-9a54-1f147f79e546","Type":"ContainerStarted","Data":"831d7ba880945af08b0e038bb2628721a350f3fe16f4e9a8fddc1aaea9b50a81"} Oct 31 12:47:51.812322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.812202 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:51.818942 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.818909 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:51.833817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:51.833764 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" podStartSLOduration=2.398332803 podStartE2EDuration="2.833749881s" totalImagesPullingTime="435.417078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:51.831289859 +0000 UTC m=+1182.646304260" watchObservedRunningTime="2026-10-31 12:47:51.833749881 +0000 UTC m=+1182.648764271" Oct 31 12:47:52.027810 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:52.027726 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:52.122519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:52.122437 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns"] Oct 31 12:47:52.125491 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:47:52.125415 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd5282b_eb99_4871_bc7f_44344133ff6d.slice/crio-f9812c3fb5e137ba6d8419c164876d1227d3755082f895c4746d355362a1b69f WatchSource:0}: Error finding container f9812c3fb5e137ba6d8419c164876d1227d3755082f895c4746d355362a1b69f: Status 404 returned error can't find the container with id f9812c3fb5e137ba6d8419c164876d1227d3755082f895c4746d355362a1b69f Oct 31 12:47:52.822386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:52.822338 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" event={"ID":"afd5282b-eb99-4871-bc7f-44344133ff6d","Type":"ContainerStarted","Data":"14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc"} Oct 31 12:47:52.822893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:52.822777 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:52.822893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:52.822821 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" event={"ID":"afd5282b-eb99-4871-bc7f-44344133ff6d","Type":"ContainerStarted","Data":"f9812c3fb5e137ba6d8419c164876d1227d3755082f895c4746d355362a1b69f"} Oct 31 12:47:52.848138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:52.848085 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" podStartSLOduration=2.37603342 podStartE2EDuration="2.84807044s" totalImagesPullingTime="472.03702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:52.844382352 +0000 UTC m=+1183.659396769" watchObservedRunningTime="2026-10-31 12:47:52.84807044 +0000 UTC m=+1183.663084869" Oct 31 12:47:52.935099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:52.935017 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:53.375636 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:53.375605 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:53.554952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:53.554888 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:53.865537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:53.864074 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:54.080235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:54.080201 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:54.359486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:54.338954 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:54.401079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:54.401037 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z"] Oct 31 12:47:54.476485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:54.475549 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz"] Oct 31 12:47:54.706022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:54.705711 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:54.860315 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:54.860287 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:55.417568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:55.417528 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:55.806752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:55.806672 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:47:56.690210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:56.690171 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:57.557860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:57.557830 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:57.828178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:57.828152 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:47:58.520010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:58.519961 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:58.838771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:58.838745 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:47:59.340914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.340873 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:59.404909 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.404781 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" podUID="20744874-84f3-4fc0-befd-26dc76141b93" containerName="payload-processing" containerID="cri-o://fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6" gracePeriod=25 Oct 31 12:47:59.476340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.476295 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" podUID="00c86f73-1646-45bc-a5b7-65d623f3aea6" containerName="payload-pre-processing" containerID="cri-o://17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36" gracePeriod=25 Oct 31 12:47:59.505292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.505266 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:59.780985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.780951 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:47:59.855512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.855444 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:47:59.896915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.896878 6727 generic.go:356] "Generic (PLEG): container finished" podID="20744874-84f3-4fc0-befd-26dc76141b93" containerID="fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6" exitCode=0 Oct 31 12:47:59.897109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.897009 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" event={"ID":"20744874-84f3-4fc0-befd-26dc76141b93","Type":"ContainerDied","Data":"fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6"} Oct 31 12:47:59.897109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.897095 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" event={"ID":"20744874-84f3-4fc0-befd-26dc76141b93","Type":"ContainerDied","Data":"977d5d18ad7de6826a17782e43f1152605c5c17e462b5b3a4a2c171213605e58"} Oct 31 12:47:59.897233 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.897125 6727 scope.go:122] "RemoveContainer" containerID="fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6" Oct 31 12:47:59.897288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.897276 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z" Oct 31 12:47:59.899865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.899842 6727 generic.go:356] "Generic (PLEG): container finished" podID="00c86f73-1646-45bc-a5b7-65d623f3aea6" containerID="17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36" exitCode=0 Oct 31 12:47:59.900024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.899893 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" Oct 31 12:47:59.900165 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.899917 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" event={"ID":"00c86f73-1646-45bc-a5b7-65d623f3aea6","Type":"ContainerDied","Data":"17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36"} Oct 31 12:47:59.900297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.900199 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz" event={"ID":"00c86f73-1646-45bc-a5b7-65d623f3aea6","Type":"ContainerDied","Data":"a4921431395539658dfd8613912393695b712eaf23674df5b196e96466ff1c5f"} Oct 31 12:47:59.910704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.910671 6727 scope.go:122] "RemoveContainer" containerID="fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6" Oct 31 12:47:59.911013 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:59.910987 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6\": container with ID starting with fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6 not found: ID does not exist" containerID="fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6" Oct 31 12:47:59.911101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.911024 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6"} err="failed to get container status \"fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6\": rpc error: code = NotFound desc = could not find container \"fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6\": container with ID starting with fe850d2cabc53ea70679088d9a09b814e0bfba97ea6e6ad15b952e29e9e0dfa6 not found: ID does not exist" Oct 31 12:47:59.911101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.911047 6727 scope.go:122] "RemoveContainer" containerID="17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36" Oct 31 12:47:59.912151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.912098 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20744874-84f3-4fc0-befd-26dc76141b93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20744874-84f3-4fc0-befd-26dc76141b93-plugins-config-volume\") pod \"20744874-84f3-4fc0-befd-26dc76141b93\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " Oct 31 12:47:59.912391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.912371 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20744874-84f3-4fc0-befd-26dc76141b93-kube-api-access-rl9vp\" (UniqueName: \"kubernetes.io/projected/20744874-84f3-4fc0-befd-26dc76141b93-kube-api-access-rl9vp\") pod \"20744874-84f3-4fc0-befd-26dc76141b93\" (UID: \"20744874-84f3-4fc0-befd-26dc76141b93\") " Oct 31 12:47:59.913855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.913829 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20744874-84f3-4fc0-befd-26dc76141b93-plugins-config-volume" pod "20744874-84f3-4fc0-befd-26dc76141b93" (UID: "20744874-84f3-4fc0-befd-26dc76141b93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:47:59.916250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.916218 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20744874-84f3-4fc0-befd-26dc76141b93-kube-api-access-rl9vp" pod "20744874-84f3-4fc0-befd-26dc76141b93" (UID: "20744874-84f3-4fc0-befd-26dc76141b93"). InnerVolumeSpecName "kube-api-access-rl9vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:47:59.925561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.925538 6727 scope.go:122] "RemoveContainer" containerID="17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36" Oct 31 12:47:59.925899 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:47:59.925873 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36\": container with ID starting with 17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36 not found: ID does not exist" containerID="17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36" Oct 31 12:47:59.925984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:47:59.925905 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36"} err="failed to get container status \"17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36\": rpc error: code = NotFound desc = could not find container \"17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36\": container with ID starting with 17f963dfaf9b797a540f21115d961a29d34bc2eb5d643205964d801c1ff9ba36 not found: ID does not exist" Oct 31 12:48:00.014477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.014437 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c86f73-1646-45bc-a5b7-65d623f3aea6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c86f73-1646-45bc-a5b7-65d623f3aea6-plugins-config-volume\") pod \"00c86f73-1646-45bc-a5b7-65d623f3aea6\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " Oct 31 12:48:00.014628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.014590 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c86f73-1646-45bc-a5b7-65d623f3aea6-kube-api-access-qh28j\" (UniqueName: \"kubernetes.io/projected/00c86f73-1646-45bc-a5b7-65d623f3aea6-kube-api-access-qh28j\") pod \"00c86f73-1646-45bc-a5b7-65d623f3aea6\" (UID: \"00c86f73-1646-45bc-a5b7-65d623f3aea6\") " Oct 31 12:48:00.014900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.014876 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c86f73-1646-45bc-a5b7-65d623f3aea6-plugins-config-volume" pod "00c86f73-1646-45bc-a5b7-65d623f3aea6" (UID: "00c86f73-1646-45bc-a5b7-65d623f3aea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:00.015228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.015209 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl9vp\" (UniqueName: \"kubernetes.io/projected/20744874-84f3-4fc0-befd-26dc76141b93-kube-api-access-rl9vp\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:00.015295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.015232 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c86f73-1646-45bc-a5b7-65d623f3aea6-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:00.015295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.015258 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20744874-84f3-4fc0-befd-26dc76141b93-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:00.017141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.017116 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c86f73-1646-45bc-a5b7-65d623f3aea6-kube-api-access-qh28j" pod "00c86f73-1646-45bc-a5b7-65d623f3aea6" (UID: "00c86f73-1646-45bc-a5b7-65d623f3aea6"). InnerVolumeSpecName "kube-api-access-qh28j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:00.116420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.116350 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh28j\" (UniqueName: \"kubernetes.io/projected/00c86f73-1646-45bc-a5b7-65d623f3aea6-kube-api-access-qh28j\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:00.226081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.226056 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z"] Oct 31 12:48:00.234148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.234128 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-711af2-6f9dd9d865-p795z"] Oct 31 12:48:00.257398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.257363 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz"] Oct 31 12:48:00.261487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.261451 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-711af2-8c547c679-rx2tz"] Oct 31 12:48:00.415322 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:48:00.409389 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:48:00.587502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:00.585965 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:01.304222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:01.304191 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c86f73-1646-45bc-a5b7-65d623f3aea6" path="/var/lib/kubelet/pods/00c86f73-1646-45bc-a5b7-65d623f3aea6/volumes" Oct 31 12:48:01.304931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:01.304909 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20744874-84f3-4fc0-befd-26dc76141b93" path="/var/lib/kubelet/pods/20744874-84f3-4fc0-befd-26dc76141b93/volumes" Oct 31 12:48:01.987663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:01.987618 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:02.992378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:02.992342 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:03.834032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:03.833985 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:04.565757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.565723 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l"] Oct 31 12:48:04.567155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.567124 6727 cpu_manager.go:414] "containerMap: removing container" podUID="20744874-84f3-4fc0-befd-26dc76141b93" containerName="payload-processing" Oct 31 12:48:04.567155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.567153 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20744874-84f3-4fc0-befd-26dc76141b93" containerName="payload-processing" Oct 31 12:48:04.567346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.567288 6727 cpu_manager.go:414] "containerMap: removing container" podUID="00c86f73-1646-45bc-a5b7-65d623f3aea6" containerName="payload-pre-processing" Oct 31 12:48:04.567346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.567297 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c86f73-1646-45bc-a5b7-65d623f3aea6" containerName="payload-pre-processing" Oct 31 12:48:04.567517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.567498 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c86f73-1646-45bc-a5b7-65d623f3aea6" containerName="payload-pre-processing" Oct 31 12:48:04.567647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.567521 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="20744874-84f3-4fc0-befd-26dc76141b93" containerName="payload-processing" Oct 31 12:48:04.575566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.575543 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.590396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.590370 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l"] Oct 31 12:48:04.591766 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.591721 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e1fc0ffe-serving-cert\"" Oct 31 12:48:04.670727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.670702 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:04.714486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.711607 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-maas-api-tls\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.714486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.711895 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.714486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.712092 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-metrics-certs\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.714486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.712510 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-kube-api-access-84gjr\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.815005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.814969 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-kube-api-access-84gjr\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.815309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.815239 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-maas-api-tls\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.815309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.815294 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.815544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.815335 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-metrics-certs\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.816457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.816388 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.818895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.818871 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-maas-api-tls\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.819055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.818957 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-metrics-certs\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.841788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.841756 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-kube-api-access-84gjr\") pod \"maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.853349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.852165 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:04.905192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.905155 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:04.964720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.964689 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j"] Oct 31 12:48:04.990695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:04.990665 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5"] Oct 31 12:48:05.114389 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:05.114356 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:05.210775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:05.210748 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l"] Oct 31 12:48:05.211284 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:05.211262 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845e130c_29bc_4e5e_a66c_e5f8ab4baf09.slice/crio-ba90e6b440fca477eafeec4329f649b6f2f4266c62e21d8b2359e21780b6ce47 WatchSource:0}: Error finding container ba90e6b440fca477eafeec4329f649b6f2f4266c62e21d8b2359e21780b6ce47: Status 404 returned error can't find the container with id ba90e6b440fca477eafeec4329f649b6f2f4266c62e21d8b2359e21780b6ce47 Oct 31 12:48:05.308520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:05.308497 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:05.984163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:05.984114 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" event={"ID":"845e130c-29bc-4e5e-a66c-e5f8ab4baf09","Type":"ContainerStarted","Data":"a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6"} Oct 31 12:48:05.984648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:05.984212 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" event={"ID":"845e130c-29bc-4e5e-a66c-e5f8ab4baf09","Type":"ContainerStarted","Data":"ba90e6b440fca477eafeec4329f649b6f2f4266c62e21d8b2359e21780b6ce47"} Oct 31 12:48:05.984648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:05.984355 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:06.073361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:06.073326 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:06.075631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:06.075583 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" podStartSLOduration=1.540053745 podStartE2EDuration="2.075569601s" totalImagesPullingTime="535.515856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:06.070941151 +0000 UTC m=+1196.885955543" watchObservedRunningTime="2026-10-31 12:48:06.075569601 +0000 UTC m=+1196.890583991" Oct 31 12:48:06.831167 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:06.831134 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:07.457359 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:07.457327 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:07.859884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:07.859838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:08.283372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.283287 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74"] Oct 31 12:48:08.290737 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.290682 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.317152 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.317124 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74"] Oct 31 12:48:08.366552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.366524 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fb2b1d-openshift-default-dockercfg-6b2h6\"" Oct 31 12:48:08.401807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.401775 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.401831 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ad52d0-9a59-4d88-8a60-90a677715a1a-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402124 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.402101 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402208 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.402159 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.402377 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402450 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.402434 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.402501 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqj6\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-kube-api-access-8zqj6\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.402579 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.402752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.402725 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504344 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504304 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504364 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504400 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ad52d0-9a59-4d88-8a60-90a677715a1a-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504440 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504504 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504718 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504753 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504777 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqj6\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-kube-api-access-8zqj6\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.504888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.504851 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.505409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.505058 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.505409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.505252 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.505409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.505353 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.505700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.505677 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.505755 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.505683 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ad52d0-9a59-4d88-8a60-90a677715a1a-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.507423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.507390 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.507579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.507504 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.517186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.517159 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqj6\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-kube-api-access-8zqj6\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.517281 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.517220 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.614812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.614776 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:08.934416 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:08.933375 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ad52d0_9a59_4d88_8a60_90a677715a1a.slice/crio-a6d19aa7a27cfc4dbe7fc0cbc53a0f22973ab6ad2b48a27bc2b0698f8b3ffda2 WatchSource:0}: Error finding container a6d19aa7a27cfc4dbe7fc0cbc53a0f22973ab6ad2b48a27bc2b0698f8b3ffda2: Status 404 returned error can't find the container with id a6d19aa7a27cfc4dbe7fc0cbc53a0f22973ab6ad2b48a27bc2b0698f8b3ffda2 Oct 31 12:48:08.934416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.933734 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74"] Oct 31 12:48:08.942887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.942088 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:08.942887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.942189 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:08.942887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.942242 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:08.984955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:08.984929 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:09.013030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:09.013002 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" event={"ID":"99ad52d0-9a59-4d88-8a60-90a677715a1a","Type":"ContainerStarted","Data":"a6d19aa7a27cfc4dbe7fc0cbc53a0f22973ab6ad2b48a27bc2b0698f8b3ffda2"} Oct 31 12:48:09.665427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:09.665380 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:09.961348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:09.961256 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:09.968484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:09.967522 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" podUID="6c87e2ff-63e2-4032-92f9-8e535fe0ce47" containerName="payload-processing" containerID="cri-o://e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61" gracePeriod=25 Oct 31 12:48:09.994735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:09.994440 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" podUID="4a8e8ba7-c995-4568-8ab6-12b61373350a" containerName="payload-pre-processing" containerID="cri-o://7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51" gracePeriod=25 Oct 31 12:48:10.044739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.044686 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" event={"ID":"99ad52d0-9a59-4d88-8a60-90a677715a1a","Type":"ContainerStarted","Data":"8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361"} Oct 31 12:48:10.147246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.128644 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" podStartSLOduration=2.128619835 podStartE2EDuration="2.128619835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:10.107287362 +0000 UTC m=+1200.922301755" watchObservedRunningTime="2026-10-31 12:48:10.128619835 +0000 UTC m=+1200.943634225" Oct 31 12:48:10.514913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.514828 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:48:10.558846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.557547 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a8e8ba7-c995-4568-8ab6-12b61373350a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8e8ba7-c995-4568-8ab6-12b61373350a-plugins-config-volume\") pod \"4a8e8ba7-c995-4568-8ab6-12b61373350a\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " Oct 31 12:48:10.558846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.557905 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8e8ba7-c995-4568-8ab6-12b61373350a-kube-api-access-s7z89\" (UniqueName: \"kubernetes.io/projected/4a8e8ba7-c995-4568-8ab6-12b61373350a-kube-api-access-s7z89\") pod \"4a8e8ba7-c995-4568-8ab6-12b61373350a\" (UID: \"4a8e8ba7-c995-4568-8ab6-12b61373350a\") " Oct 31 12:48:10.559371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.559348 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8e8ba7-c995-4568-8ab6-12b61373350a-plugins-config-volume" pod "4a8e8ba7-c995-4568-8ab6-12b61373350a" (UID: "4a8e8ba7-c995-4568-8ab6-12b61373350a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:10.561003 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.560977 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8e8ba7-c995-4568-8ab6-12b61373350a-kube-api-access-s7z89" pod "4a8e8ba7-c995-4568-8ab6-12b61373350a" (UID: "4a8e8ba7-c995-4568-8ab6-12b61373350a"). InnerVolumeSpecName "kube-api-access-s7z89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:10.617287 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.617261 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:10.617430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.617350 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 12:48:10.617430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.617402 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 12:48:10.651202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.648316 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:48:10.660261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.660234 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7z89\" (UniqueName: \"kubernetes.io/projected/4a8e8ba7-c995-4568-8ab6-12b61373350a-kube-api-access-s7z89\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:10.660354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.660263 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a8e8ba7-c995-4568-8ab6-12b61373350a-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:10.829542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.790442 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-plugins-config-volume\") pod \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " Oct 31 12:48:10.850484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.830259 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-kube-api-access-cm9l8\" (UniqueName: \"kubernetes.io/projected/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-kube-api-access-cm9l8\") pod \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\" (UID: \"6c87e2ff-63e2-4032-92f9-8e535fe0ce47\") " Oct 31 12:48:10.850729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.806714 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-plugins-config-volume" pod "6c87e2ff-63e2-4032-92f9-8e535fe0ce47" (UID: "6c87e2ff-63e2-4032-92f9-8e535fe0ce47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:10.974490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:10.965677 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-kube-api-access-cm9l8" pod "6c87e2ff-63e2-4032-92f9-8e535fe0ce47" (UID: "6c87e2ff-63e2-4032-92f9-8e535fe0ce47"). InnerVolumeSpecName "kube-api-access-cm9l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:11.117761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.015579 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:11.125479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.122770 6727 generic.go:356] "Generic (PLEG): container finished" podID="4a8e8ba7-c995-4568-8ab6-12b61373350a" containerID="7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51" exitCode=0 Oct 31 12:48:11.125479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.123010 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" event={"ID":"4a8e8ba7-c995-4568-8ab6-12b61373350a","Type":"ContainerDied","Data":"7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51"} Oct 31 12:48:11.125479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.123076 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" event={"ID":"4a8e8ba7-c995-4568-8ab6-12b61373350a","Type":"ContainerDied","Data":"c382fd8ca315501fb5f83b9f402a1882acea7313ce51b63b0297b5e0f3009033"} Oct 31 12:48:11.125479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.123123 6727 scope.go:122] "RemoveContainer" containerID="7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51" Oct 31 12:48:11.125479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.123313 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" Oct 31 12:48:11.127205 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.126967 6727 generic.go:356] "Generic (PLEG): container finished" podID="6c87e2ff-63e2-4032-92f9-8e535fe0ce47" containerID="e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61" exitCode=0 Oct 31 12:48:11.127205 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.126994 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" event={"ID":"6c87e2ff-63e2-4032-92f9-8e535fe0ce47","Type":"ContainerDied","Data":"e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61"} Oct 31 12:48:11.127205 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.127018 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" Oct 31 12:48:11.127205 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.127040 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" event={"ID":"6c87e2ff-63e2-4032-92f9-8e535fe0ce47","Type":"ContainerDied","Data":"b95462af72ae53dbe51196f260a0006704ef3102dafb168fa0f5a479af6e863b"} Oct 31 12:48:11.144204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.144173 6727 scope.go:122] "RemoveContainer" containerID="7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51" Oct 31 12:48:11.144605 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:48:11.144584 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51\": container with ID starting with 7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51 not found: ID does not exist" containerID="7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51" Oct 31 12:48:11.144675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.144616 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51"} err="failed to get container status \"7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51\": rpc error: code = NotFound desc = could not find container \"7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51\": container with ID starting with 7cc77efb34eb4f70beb28d95fe9617790f6306448c41447cf0ef6ada434b2b51 not found: ID does not exist" Oct 31 12:48:11.144675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.144634 6727 scope.go:122] "RemoveContainer" containerID="e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61" Oct 31 12:48:11.155923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.155339 6727 scope.go:122] "RemoveContainer" containerID="e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61" Oct 31 12:48:11.156710 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:48:11.156682 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61\": container with ID starting with e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61 not found: ID does not exist" containerID="e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61" Oct 31 12:48:11.156793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.156720 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61"} err="failed to get container status \"e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61\": rpc error: code = NotFound desc = could not find container \"e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61\": container with ID starting with e637b680fb135f1856648a053f8c5bff11ceb24aa35b22015fde1c8497ddac61 not found: ID does not exist" Oct 31 12:48:11.173509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.162912 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5"] Oct 31 12:48:11.207483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.205666 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm9l8\" (UniqueName: \"kubernetes.io/projected/6c87e2ff-63e2-4032-92f9-8e535fe0ce47-kube-api-access-cm9l8\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:11.213520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.212145 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5"] Oct 31 12:48:11.231180 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.231147 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j"] Oct 31 12:48:11.232509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.232434 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j"] Oct 31 12:48:11.305556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.302245 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:11.380770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.380674 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8e8ba7-c995-4568-8ab6-12b61373350a" path="/var/lib/kubelet/pods/4a8e8ba7-c995-4568-8ab6-12b61373350a/volumes" Oct 31 12:48:11.381876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.381851 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c87e2ff-63e2-4032-92f9-8e535fe0ce47" path="/var/lib/kubelet/pods/6c87e2ff-63e2-4032-92f9-8e535fe0ce47/volumes" Oct 31 12:48:11.436798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.436757 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-bb3cf5-68fffcb979-cm66j" podUID="6c87e2ff-63e2-4032-92f9-8e535fe0ce47" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.111:50052\" within 1s: context deadline exceeded" Oct 31 12:48:11.437099 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:11.437081 6727 logging.go:55] [core] [Channel #928 SubChannel #929]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.111:50052", ServerName: "10.134.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.111:50052: operation was canceled" Oct 31 12:48:11.437251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.437214 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bb3cf5-674484f8c5-lh7l5" podUID="4a8e8ba7-c995-4568-8ab6-12b61373350a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.112:50052\" within 1s: context deadline exceeded" Oct 31 12:48:11.437359 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:11.437342 6727 logging.go:55] [core] [Channel #926 SubChannel #927]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.112:50052", ServerName: "10.134.0.112:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.112:50052: operation was canceled" Oct 31 12:48:11.504786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.504646 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:11.617487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.617236 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 12:48:11.617487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.617307 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 12:48:11.998544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:11.998508 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:12.615844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:12.615807 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 12:48:12.616118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:12.616089 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 12:48:13.203372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:13.203304 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:13.423127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:13.422641 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:13.615993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:13.615960 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 12:48:13.616191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:13.616012 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 12:48:14.255913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:14.255874 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:14.541420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:14.541326 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:14.615435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:14.615401 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 12:48:14.615620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:14.615498 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 12:48:14.683486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:14.679804 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:15.028762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:15.028737 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:15.218227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:15.218152 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:15.367867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:15.367826 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:15.687355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:15.686138 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:15.687860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:15.687785 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:15.689318 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:15.689080 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:48:16.088050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:16.088000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:16.279600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:16.279569 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:17.151711 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:17.151669 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:17.459952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:17.459863 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:17.654555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:17.654521 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:18.154726 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:18.154689 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:19.100181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:19.100125 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:20.074437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:20.074401 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:20.712659 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:20.712623 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf"] Oct 31 12:48:20.719134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:20.719103 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" podUID="64ea685b-ff9d-4e0d-ae04-a390e74159d0" containerName="maas-api" containerID="cri-o://219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940" gracePeriod=30 Oct 31 12:48:20.800336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:20.800310 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:21.045743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.045532 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:21.227335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.227300 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:21.304336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.304272 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:48:21.338814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.338774 6727 generic.go:356] "Generic (PLEG): container finished" podID="64ea685b-ff9d-4e0d-ae04-a390e74159d0" containerID="219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940" exitCode=0 Oct 31 12:48:21.338992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.338832 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" Oct 31 12:48:21.338992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.338857 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" event={"ID":"64ea685b-ff9d-4e0d-ae04-a390e74159d0","Type":"ContainerDied","Data":"219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940"} Oct 31 12:48:21.338992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.338920 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf" event={"ID":"64ea685b-ff9d-4e0d-ae04-a390e74159d0","Type":"ContainerDied","Data":"ccd8d65b1dcf6574960cf63816811fdb3af6f42a082692b9adcf966807f5afc5"} Oct 31 12:48:21.338992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.338949 6727 scope.go:122] "RemoveContainer" containerID="219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940" Oct 31 12:48:21.370817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.370795 6727 scope.go:122] "RemoveContainer" containerID="219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940" Oct 31 12:48:21.371265 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:48:21.371239 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940\": container with ID starting with 219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940 not found: ID does not exist" containerID="219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940" Oct 31 12:48:21.371373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.371276 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940"} err="failed to get container status \"219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940\": rpc error: code = NotFound desc = could not find container \"219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940\": container with ID starting with 219ed8c2499d91e39e96f8cb17aea5da0af55f11f5a35be157149e6f356b6940 not found: ID does not exist" Oct 31 12:48:21.389839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.389799 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ea685b-ff9d-4e0d-ae04-a390e74159d0-kube-api-access-pmhv8\" (UniqueName: \"kubernetes.io/projected/64ea685b-ff9d-4e0d-ae04-a390e74159d0-kube-api-access-pmhv8\") pod \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " Oct 31 12:48:21.390068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.389885 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-maas-api-tls\") pod \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " Oct 31 12:48:21.390068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.390056 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ea685b-ff9d-4e0d-ae04-a390e74159d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ea685b-ff9d-4e0d-ae04-a390e74159d0-openshift-service-ca\") pod \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " Oct 31 12:48:21.390192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.390115 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-metrics-certs\") pod \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\" (UID: \"64ea685b-ff9d-4e0d-ae04-a390e74159d0\") " Oct 31 12:48:21.391609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.391580 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ea685b-ff9d-4e0d-ae04-a390e74159d0-openshift-service-ca" pod "64ea685b-ff9d-4e0d-ae04-a390e74159d0" (UID: "64ea685b-ff9d-4e0d-ae04-a390e74159d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:21.393123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.393075 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-maas-api-tls" pod "64ea685b-ff9d-4e0d-ae04-a390e74159d0" (UID: "64ea685b-ff9d-4e0d-ae04-a390e74159d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:21.398979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.398944 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-metrics-certs" pod "64ea685b-ff9d-4e0d-ae04-a390e74159d0" (UID: "64ea685b-ff9d-4e0d-ae04-a390e74159d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:21.398979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.398956 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ea685b-ff9d-4e0d-ae04-a390e74159d0-kube-api-access-pmhv8" pod "64ea685b-ff9d-4e0d-ae04-a390e74159d0" (UID: "64ea685b-ff9d-4e0d-ae04-a390e74159d0"). InnerVolumeSpecName "kube-api-access-pmhv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:21.492689 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.492658 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ea685b-ff9d-4e0d-ae04-a390e74159d0-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:21.492689 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.492687 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:21.492689 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.492696 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhv8\" (UniqueName: \"kubernetes.io/projected/64ea685b-ff9d-4e0d-ae04-a390e74159d0-kube-api-access-pmhv8\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:21.492947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.492708 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ea685b-ff9d-4e0d-ae04-a390e74159d0-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:21.529577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.529543 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:21.785488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.784126 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:21.856484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.847509 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf"] Oct 31 12:48:21.874245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:21.867053 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-711af2-646fd7d9cb-7nhsf"] Oct 31 12:48:22.021627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.021509 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:22.945106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.945058 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:22.996268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.996233 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg"] Oct 31 12:48:22.997196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997174 6727 cpu_manager.go:414] "containerMap: removing container" podUID="6c87e2ff-63e2-4032-92f9-8e535fe0ce47" containerName="payload-processing" Oct 31 12:48:22.997196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997196 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c87e2ff-63e2-4032-92f9-8e535fe0ce47" containerName="payload-processing" Oct 31 12:48:22.997385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997241 6727 cpu_manager.go:414] "containerMap: removing container" podUID="4a8e8ba7-c995-4568-8ab6-12b61373350a" containerName="payload-pre-processing" Oct 31 12:48:22.997385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997247 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8e8ba7-c995-4568-8ab6-12b61373350a" containerName="payload-pre-processing" Oct 31 12:48:22.997385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997275 6727 cpu_manager.go:414] "containerMap: removing container" podUID="64ea685b-ff9d-4e0d-ae04-a390e74159d0" containerName="maas-api" Oct 31 12:48:22.997385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997281 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ea685b-ff9d-4e0d-ae04-a390e74159d0" containerName="maas-api" Oct 31 12:48:22.997554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997409 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8e8ba7-c995-4568-8ab6-12b61373350a" containerName="payload-pre-processing" Oct 31 12:48:22.997554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997423 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ea685b-ff9d-4e0d-ae04-a390e74159d0" containerName="maas-api" Oct 31 12:48:22.997554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:22.997432 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c87e2ff-63e2-4032-92f9-8e535fe0ce47" containerName="payload-processing" Oct 31 12:48:23.002971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.002953 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.017491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.017447 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ad1af7\"" Oct 31 12:48:23.017755 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.017737 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ad1af7-dockercfg-7trlr\"" Oct 31 12:48:23.023734 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.023700 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg"] Oct 31 12:48:23.028418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.028391 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj"] Oct 31 12:48:23.035943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.035920 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.049971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.049940 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj"] Oct 31 12:48:23.092035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.092001 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc0a033-4793-43ba-bb55-a960b69cf7dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.092263 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.092227 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6fm\" (UniqueName: \"kubernetes.io/projected/bdc0a033-4793-43ba-bb55-a960b69cf7dc-kube-api-access-sr6fm\") pod \"payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.129670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.129634 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:23.197542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.197240 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6fm\" (UniqueName: \"kubernetes.io/projected/bdc0a033-4793-43ba-bb55-a960b69cf7dc-kube-api-access-sr6fm\") pod \"payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.199489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.197724 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22db\" (UniqueName: \"kubernetes.io/projected/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-kube-api-access-b22db\") pod \"payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.199489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.197767 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc0a033-4793-43ba-bb55-a960b69cf7dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.199489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.197814 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.230489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.208786 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc0a033-4793-43ba-bb55-a960b69cf7dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.246484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.245500 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6fm\" (UniqueName: \"kubernetes.io/projected/bdc0a033-4793-43ba-bb55-a960b69cf7dc-kube-api-access-sr6fm\") pod \"payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.305487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.300006 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22db\" (UniqueName: \"kubernetes.io/projected/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-kube-api-access-b22db\") pod \"payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.305487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.300059 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.305487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.301037 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.317486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.315669 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ea685b-ff9d-4e0d-ae04-a390e74159d0" path="/var/lib/kubelet/pods/64ea685b-ff9d-4e0d-ae04-a390e74159d0/volumes" Oct 31 12:48:23.340490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.322818 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:23.340490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.324195 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22db\" (UniqueName: \"kubernetes.io/projected/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-kube-api-access-b22db\") pod \"payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.366496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.363155 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:23.795822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.795721 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg"] Oct 31 12:48:23.795956 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:23.795866 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc0a033_4793_43ba_bb55_a960b69cf7dc.slice/crio-beab81599443f55d45d6ab86fd545322a9ccb0e8b58da91c8d1c46d81ef023c5 WatchSource:0}: Error finding container beab81599443f55d45d6ab86fd545322a9ccb0e8b58da91c8d1c46d81ef023c5: Status 404 returned error can't find the container with id beab81599443f55d45d6ab86fd545322a9ccb0e8b58da91c8d1c46d81ef023c5 Oct 31 12:48:23.822836 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:23.822808 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj"] Oct 31 12:48:23.825687 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:23.825657 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb6e92d_9d0a_41fc_bd1f_8ace2848716e.slice/crio-3d7c9ad52e22e764ffcf6dfc253fe3c012919755ea5677cc40a7d28921a556d0 WatchSource:0}: Error finding container 3d7c9ad52e22e764ffcf6dfc253fe3c012919755ea5677cc40a7d28921a556d0: Status 404 returned error can't find the container with id 3d7c9ad52e22e764ffcf6dfc253fe3c012919755ea5677cc40a7d28921a556d0 Oct 31 12:48:24.297176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.297140 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:24.372606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.372574 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" event={"ID":"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e","Type":"ContainerStarted","Data":"2f680eaa7365b01b405e29d4471e61cab0b3baa1ad37dba1476a43e2604eb75b"} Oct 31 12:48:24.372801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.372637 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" event={"ID":"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e","Type":"ContainerStarted","Data":"3d7c9ad52e22e764ffcf6dfc253fe3c012919755ea5677cc40a7d28921a556d0"} Oct 31 12:48:24.372880 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.372806 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:24.374555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.374531 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" event={"ID":"bdc0a033-4793-43ba-bb55-a960b69cf7dc","Type":"ContainerStarted","Data":"5dc66d2ce2f8c61da45370ce3c7d2986d4eb54d4a3951fd899a7a9a4d5055fc2"} Oct 31 12:48:24.374662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.374573 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" event={"ID":"bdc0a033-4793-43ba-bb55-a960b69cf7dc","Type":"ContainerStarted","Data":"beab81599443f55d45d6ab86fd545322a9ccb0e8b58da91c8d1c46d81ef023c5"} Oct 31 12:48:24.374662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.374629 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:24.403858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.403816 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" podStartSLOduration=2.403804312 podStartE2EDuration="2.403804312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:24.400015376 +0000 UTC m=+1215.215029770" watchObservedRunningTime="2026-10-31 12:48:24.403804312 +0000 UTC m=+1215.218818702" Oct 31 12:48:24.436313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:24.436265 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" podStartSLOduration=2.4362548 podStartE2EDuration="2.4362548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:24.431314231 +0000 UTC m=+1215.246328621" watchObservedRunningTime="2026-10-31 12:48:24.4362548 +0000 UTC m=+1215.251269191" Oct 31 12:48:25.598190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:25.598154 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:25.836482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:25.824303 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:27.058707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:27.052777 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:28.016479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.016431 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475"] Oct 31 12:48:28.017923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.017892 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:28.021533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.021509 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw"] Oct 31 12:48:28.021728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.021543 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.026584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.026563 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.088839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.088812 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw"] Oct 31 12:48:28.094855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.094833 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475"] Oct 31 12:48:28.102805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.102785 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e43d8b\"" Oct 31 12:48:28.148621 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.148583 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572h4\" (UniqueName: \"kubernetes.io/projected/69719755-33c7-41bd-9ec1-1feadbf10a47-kube-api-access-572h4\") pod \"payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.148805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.148777 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69719755-33c7-41bd-9ec1-1feadbf10a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.149030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.149007 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5h5\" (UniqueName: \"kubernetes.io/projected/573a12f7-4680-46bc-83f4-d65080d436ad-kube-api-access-hr5h5\") pod \"payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.149204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.149186 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573a12f7-4680-46bc-83f4-d65080d436ad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.255900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.255859 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572h4\" (UniqueName: \"kubernetes.io/projected/69719755-33c7-41bd-9ec1-1feadbf10a47-kube-api-access-572h4\") pod \"payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.256958 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.256845 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69719755-33c7-41bd-9ec1-1feadbf10a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.257329 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.257300 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5h5\" (UniqueName: \"kubernetes.io/projected/573a12f7-4680-46bc-83f4-d65080d436ad-kube-api-access-hr5h5\") pod \"payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.257777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.257753 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573a12f7-4680-46bc-83f4-d65080d436ad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.262581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.262550 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69719755-33c7-41bd-9ec1-1feadbf10a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.263040 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.256243 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:28.275536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.270272 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573a12f7-4680-46bc-83f4-d65080d436ad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.292522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.287225 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-572h4\" (UniqueName: \"kubernetes.io/projected/69719755-33c7-41bd-9ec1-1feadbf10a47-kube-api-access-572h4\") pod \"payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.304261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.304231 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5h5\" (UniqueName: \"kubernetes.io/projected/573a12f7-4680-46bc-83f4-d65080d436ad-kube-api-access-hr5h5\") pod \"payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.349498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.339893 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:28.350295 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.350274 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:28.502757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.502725 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:28.865017 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:28.864988 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573a12f7_4680_46bc_83f4_d65080d436ad.slice/crio-3a1e215f6c749ae457b931420fe1af74b06afd1a58bc3bd8ca49c90c738feb22 WatchSource:0}: Error finding container 3a1e215f6c749ae457b931420fe1af74b06afd1a58bc3bd8ca49c90c738feb22: Status 404 returned error can't find the container with id 3a1e215f6c749ae457b931420fe1af74b06afd1a58bc3bd8ca49c90c738feb22 Oct 31 12:48:28.865978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.865950 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw"] Oct 31 12:48:28.958117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:28.958088 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475"] Oct 31 12:48:28.958900 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:28.958879 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69719755_33c7_41bd_9ec1_1feadbf10a47.slice/crio-805b3cc17b0072df0299982f8b2b184ef69e6c0c487af0283257ad09154fe377 WatchSource:0}: Error finding container 805b3cc17b0072df0299982f8b2b184ef69e6c0c487af0283257ad09154fe377: Status 404 returned error can't find the container with id 805b3cc17b0072df0299982f8b2b184ef69e6c0c487af0283257ad09154fe377 Oct 31 12:48:29.595164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.595127 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" event={"ID":"573a12f7-4680-46bc-83f4-d65080d436ad","Type":"ContainerStarted","Data":"d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4"} Oct 31 12:48:29.595789 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.595191 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" event={"ID":"573a12f7-4680-46bc-83f4-d65080d436ad","Type":"ContainerStarted","Data":"3a1e215f6c749ae457b931420fe1af74b06afd1a58bc3bd8ca49c90c738feb22"} Oct 31 12:48:29.596418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.596393 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:29.599138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.599112 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" event={"ID":"69719755-33c7-41bd-9ec1-1feadbf10a47","Type":"ContainerStarted","Data":"4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c"} Oct 31 12:48:29.599346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.599321 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:29.599455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.599356 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" event={"ID":"69719755-33c7-41bd-9ec1-1feadbf10a47","Type":"ContainerStarted","Data":"805b3cc17b0072df0299982f8b2b184ef69e6c0c487af0283257ad09154fe377"} Oct 31 12:48:29.606842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.606818 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:29.632001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.631428 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" podStartSLOduration=2.631410094 podStartE2EDuration="2.631410094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:29.626695456 +0000 UTC m=+1220.441709849" watchObservedRunningTime="2026-10-31 12:48:29.631410094 +0000 UTC m=+1220.446424482" Oct 31 12:48:29.655564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:29.650868 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" podStartSLOduration=2.6508492820000003 podStartE2EDuration="2.650849282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:29.648225561 +0000 UTC m=+1220.463239954" watchObservedRunningTime="2026-10-31 12:48:29.650849282 +0000 UTC m=+1220.465863670" Oct 31 12:48:30.345438 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:48:30.345402 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:48:30.559883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:30.559846 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:31.027317 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:31.027283 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:31.372338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:31.372308 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:31.596692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:31.596661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:32.229681 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:32.229641 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:32.576238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:32.576206 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:32.872067 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:32.871995 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:33.161283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:33.161195 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9"] Oct 31 12:48:33.362489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:33.355453 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:33.558103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:33.558029 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:33.700008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:33.699971 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:48:33.756618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:33.756579 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:33.903458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:33.903424 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:34.246675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:34.246596 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:34.829245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:34.829201 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:35.449514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:35.447607 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:48:35.449514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:35.447696 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:48:36.920373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:36.920335 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:37.092264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:37.092231 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:37.659656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:37.659611 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:37.737607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:37.737563 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:37.740037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:37.740004 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:37.741365 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:37.741339 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:48:37.795623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:37.795579 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:37.991895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:37.991798 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:38.053440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.053398 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:48:38.103597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.103551 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:48:38.162367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.162306 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" podUID="e21586c5-6d0e-4ded-9809-270fd1022e95" containerName="payload-processing" containerID="cri-o://fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113" gracePeriod=25 Oct 31 12:48:38.170539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.169670 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:38.182496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.180590 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:38.384488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.380816 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:38.447489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.446538 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:38.907538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.891152 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:38.907933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.895557 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:38.908419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.908395 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:48:38.947708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.909148 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 12:48:38.948552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.917695 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:48:38.949217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.917723 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:48:38.949752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.917742 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:48:38.950246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.948769 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:48:38.950579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.950563 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:48:38.960739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.960720 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:48:38.968644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:38.968627 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:48:39.156821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.153888 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:48:39.178580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.161763 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:48:39.178580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.162518 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:39.178580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.163038 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:48:39.178580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.172557 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:48:39.186560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.179907 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:48:39.186560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.181689 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:39.186560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.181784 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:48:39.249568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.234243 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:39.249568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.237330 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:48:39.249568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.237847 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:48:39.249568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.245078 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:48:39.249568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.245132 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:39.304612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.290084 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:48:39.386586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.385610 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:39.453171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.453049 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:39.861357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.861324 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:39.936580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.936551 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:48:39.936886 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.936872 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:48:39.937048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.937037 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:48:39.937189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.937178 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:48:39.976302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.976273 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:48:39.976665 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.976626 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:40.001724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:39.953451 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:48:40.024566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.013588 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:48:40.074157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.074126 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-897xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:48:40.074440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.074404 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:40.139538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.139444 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:48:40.155983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.155758 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:40.188496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.174808 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:48:40.231119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.231090 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:48:40.231438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.231419 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:48:40.231617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.189275 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:40.231841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.189890 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:48:40.259764 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.257666 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:48:40.288488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.273851 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:48:40.324577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.324332 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21586c5-6d0e-4ded-9809-270fd1022e95-kube-api-access-759rj\" (UniqueName: \"kubernetes.io/projected/e21586c5-6d0e-4ded-9809-270fd1022e95-kube-api-access-759rj\") pod \"e21586c5-6d0e-4ded-9809-270fd1022e95\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " Oct 31 12:48:40.324956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.324930 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21586c5-6d0e-4ded-9809-270fd1022e95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21586c5-6d0e-4ded-9809-270fd1022e95-plugins-config-volume\") pod \"e21586c5-6d0e-4ded-9809-270fd1022e95\" (UID: \"e21586c5-6d0e-4ded-9809-270fd1022e95\") " Oct 31 12:48:40.326599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.326575 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21586c5-6d0e-4ded-9809-270fd1022e95-plugins-config-volume" pod "e21586c5-6d0e-4ded-9809-270fd1022e95" (UID: "e21586c5-6d0e-4ded-9809-270fd1022e95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:40.356503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.351411 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:48:40.356914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.356686 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21586c5-6d0e-4ded-9809-270fd1022e95-kube-api-access-759rj" pod "e21586c5-6d0e-4ded-9809-270fd1022e95" (UID: "e21586c5-6d0e-4ded-9809-270fd1022e95"). InnerVolumeSpecName "kube-api-access-759rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:40.364487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.361983 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:48:40.392687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.392599 6727 generic.go:356] "Generic (PLEG): container finished" podID="e21586c5-6d0e-4ded-9809-270fd1022e95" containerID="fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113" exitCode=0 Oct 31 12:48:40.433476 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.433416 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759rj\" (UniqueName: \"kubernetes.io/projected/e21586c5-6d0e-4ded-9809-270fd1022e95-kube-api-access-759rj\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:40.433703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.433682 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21586c5-6d0e-4ded-9809-270fd1022e95-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:40.433818 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.393530 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" event={"ID":"e21586c5-6d0e-4ded-9809-270fd1022e95","Type":"ContainerDied","Data":"fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113"} Oct 31 12:48:40.433970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.393506 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" Oct 31 12:48:40.436972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.433955 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9" event={"ID":"e21586c5-6d0e-4ded-9809-270fd1022e95","Type":"ContainerDied","Data":"cafc654e8465e0ff8f541c9712090277d066e49190f64eb97c16eedf2f7fe6d8"} Oct 31 12:48:40.437087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.437028 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:48:40.437657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.433975 6727 scope.go:122] "RemoveContainer" containerID="fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113" Oct 31 12:48:40.480687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.471843 6727 scope.go:122] "RemoveContainer" containerID="fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113" Oct 31 12:48:40.493187 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:48:40.491957 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113\": container with ID starting with fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113 not found: ID does not exist" containerID="fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113" Oct 31 12:48:40.493187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.491999 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113"} err="failed to get container status \"fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113\": rpc error: code = NotFound desc = could not find container \"fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113\": container with ID starting with fa38d3df3b326187f4b7d536007bcadf75308752c933138d1a207b10cffd6113 not found: ID does not exist" Oct 31 12:48:40.618078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:40.618052 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:48:41.042060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:41.042018 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9"] Oct 31 12:48:41.051140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:41.051104 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5815-69bdd69489-7j9g9"] Oct 31 12:48:41.307203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:41.307130 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21586c5-6d0e-4ded-9809-270fd1022e95" path="/var/lib/kubelet/pods/e21586c5-6d0e-4ded-9809-270fd1022e95/volumes" Oct 31 12:48:41.689893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:41.689865 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:48:41.915895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:41.915852 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:42.954630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:42.954539 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:44.002877 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:44.002755 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:44.576119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:44.576090 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:44.753648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:44.753598 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:44.953018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:44.952931 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:45.122226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:45.122140 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:45.650097 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:45.650063 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:46.227122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:46.226883 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:46.658394 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:46.658356 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:46.920557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:46.920452 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:47.110146 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:47.110108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:47.181746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:47.181640 6727 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:48:47.193735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:47.181740 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:48:47.378978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:47.378944 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:47.591182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:47.590944 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:48.146648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:48.146617 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:48.553870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:48.553795 6727 generic.go:356] "Generic (PLEG): container finished" podID="7f585440-9878-4540-b806-6704375dc496" containerID="58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120" exitCode=0 Oct 31 12:48:48.554235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:48.553864 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" event={"ID":"7f585440-9878-4540-b806-6704375dc496","Type":"ContainerDied","Data":"58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120"} Oct 31 12:48:48.554280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:48.554235 6727 scope.go:122] "RemoveContainer" containerID="58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120" Oct 31 12:48:48.556005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:48.555976 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:48.556087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:48.556046 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:48.556123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:48.556084 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:49.187059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.187022 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:49.397419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.397391 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:49.565779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.565703 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" event={"ID":"7f585440-9878-4540-b806-6704375dc496","Type":"ContainerStarted","Data":"b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc"} Oct 31 12:48:49.803277 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.803245 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l"] Oct 31 12:48:49.803481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.803447 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" podUID="845e130c-29bc-4e5e-a66c-e5f8ab4baf09" containerName="maas-api" containerID="cri-o://a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6" gracePeriod=30 Oct 31 12:48:49.920927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.920793 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:49.973223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.971108 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:48:49.973223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.971258 6727 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 31 12:48:49.973223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:49.971302 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 31 12:48:50.618241 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.618213 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:50.647420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.647130 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-metrics-certs\") pod \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " Oct 31 12:48:50.647420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.647333 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-openshift-service-ca\") pod \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " Oct 31 12:48:50.647420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.647371 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-maas-api-tls\") pod \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " Oct 31 12:48:50.647420 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.647429 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-kube-api-access-84gjr\") pod \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\" (UID: \"845e130c-29bc-4e5e-a66c-e5f8ab4baf09\") " Oct 31 12:48:50.652168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.651998 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-kube-api-access-84gjr" pod "845e130c-29bc-4e5e-a66c-e5f8ab4baf09" (UID: "845e130c-29bc-4e5e-a66c-e5f8ab4baf09"). InnerVolumeSpecName "kube-api-access-84gjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:50.653681 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.653581 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-openshift-service-ca" pod "845e130c-29bc-4e5e-a66c-e5f8ab4baf09" (UID: "845e130c-29bc-4e5e-a66c-e5f8ab4baf09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:50.655753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.655727 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-maas-api-tls" pod "845e130c-29bc-4e5e-a66c-e5f8ab4baf09" (UID: "845e130c-29bc-4e5e-a66c-e5f8ab4baf09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:50.657985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.657960 6727 generic.go:356] "Generic (PLEG): container finished" podID="845e130c-29bc-4e5e-a66c-e5f8ab4baf09" containerID="a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6" exitCode=0 Oct 31 12:48:50.659638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.659618 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" Oct 31 12:48:50.660086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.660065 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" event={"ID":"845e130c-29bc-4e5e-a66c-e5f8ab4baf09","Type":"ContainerDied","Data":"a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6"} Oct 31 12:48:50.681599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.661606 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l" event={"ID":"845e130c-29bc-4e5e-a66c-e5f8ab4baf09","Type":"ContainerDied","Data":"ba90e6b440fca477eafeec4329f649b6f2f4266c62e21d8b2359e21780b6ce47"} Oct 31 12:48:50.681599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.661672 6727 scope.go:122] "RemoveContainer" containerID="a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6" Oct 31 12:48:50.704453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.704305 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:50.706179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.706147 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-metrics-certs" pod "845e130c-29bc-4e5e-a66c-e5f8ab4baf09" (UID: "845e130c-29bc-4e5e-a66c-e5f8ab4baf09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:50.718790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.718747 6727 scope.go:122] "RemoveContainer" containerID="a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6" Oct 31 12:48:50.719078 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:48:50.719052 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6\": container with ID starting with a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6 not found: ID does not exist" containerID="a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6" Oct 31 12:48:50.719161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.719091 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6"} err="failed to get container status \"a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6\": rpc error: code = NotFound desc = could not find container \"a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6\": container with ID starting with a352ea3c81b4c56f83cc5a5e1d65f762ea4797c83065e13c88633c7f1e9837f6 not found: ID does not exist" Oct 31 12:48:50.750870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.750845 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:50.750870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.750865 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:50.750870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.750875 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:50.751061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.750884 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/845e130c-29bc-4e5e-a66c-e5f8ab4baf09-kube-api-access-84gjr\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:50.851733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:50.851706 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:51.001539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.000930 6727 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 31 12:48:51.001539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.001062 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 31 12:48:51.029996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.029966 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:51.035266 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.035237 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l"] Oct 31 12:48:51.039614 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.039594 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fc0ffe-566597686d-qhd9l"] Oct 31 12:48:51.252557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.252478 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:51.305865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.305832 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845e130c-29bc-4e5e-a66c-e5f8ab4baf09" path="/var/lib/kubelet/pods/845e130c-29bc-4e5e-a66c-e5f8ab4baf09/volumes" Oct 31 12:48:51.447905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.447873 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:51.991484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.991140 6727 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 31 12:48:51.991484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:51.991198 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 31 12:48:52.193498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.190128 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:52.588101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.588056 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:52.615325 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.615296 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc"] Oct 31 12:48:52.616254 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.616235 6727 cpu_manager.go:414] "containerMap: removing container" podUID="e21586c5-6d0e-4ded-9809-270fd1022e95" containerName="payload-processing" Oct 31 12:48:52.616313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.616256 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21586c5-6d0e-4ded-9809-270fd1022e95" containerName="payload-processing" Oct 31 12:48:52.616313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.616281 6727 cpu_manager.go:414] "containerMap: removing container" podUID="845e130c-29bc-4e5e-a66c-e5f8ab4baf09" containerName="maas-api" Oct 31 12:48:52.616313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.616287 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845e130c-29bc-4e5e-a66c-e5f8ab4baf09" containerName="maas-api" Oct 31 12:48:52.616424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.616413 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21586c5-6d0e-4ded-9809-270fd1022e95" containerName="payload-processing" Oct 31 12:48:52.616499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.616430 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="845e130c-29bc-4e5e-a66c-e5f8ab4baf09" containerName="maas-api" Oct 31 12:48:52.620912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.620896 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.630516 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.630495 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs"] Oct 31 12:48:52.632625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.632591 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e98df8\"" Oct 31 12:48:52.635278 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.635257 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:52.638935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.638911 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e98df8-dockercfg-tcnx7\"" Oct 31 12:48:52.641565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.641541 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc"] Oct 31 12:48:52.655941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.655918 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs"] Oct 31 12:48:52.719701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.718688 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8bt\" (UniqueName: \"kubernetes.io/projected/b7409e4f-0b82-4884-ac1e-56a382f181bf-kube-api-access-pz8bt\") pod \"payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.719701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.718833 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7409e4f-0b82-4884-ac1e-56a382f181bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.719701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.719099 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phl7s\" (UniqueName: \"kubernetes.io/projected/7b2d4864-c55c-4861-9b7e-24807e02d842-kube-api-access-phl7s\") pod \"payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:52.719701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.719187 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2d4864-c55c-4861-9b7e-24807e02d842-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:52.803486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.783315 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:52.834363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.822067 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8bt\" (UniqueName: \"kubernetes.io/projected/b7409e4f-0b82-4884-ac1e-56a382f181bf-kube-api-access-pz8bt\") pod \"payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.834363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.822190 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7409e4f-0b82-4884-ac1e-56a382f181bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.834363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.822361 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phl7s\" (UniqueName: \"kubernetes.io/projected/7b2d4864-c55c-4861-9b7e-24807e02d842-kube-api-access-phl7s\") pod \"payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:52.834363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.822510 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2d4864-c55c-4861-9b7e-24807e02d842-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:52.834363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.823537 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2d4864-c55c-4861-9b7e-24807e02d842-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:52.877536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.862765 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8bt\" (UniqueName: \"kubernetes.io/projected/b7409e4f-0b82-4884-ac1e-56a382f181bf-kube-api-access-pz8bt\") pod \"payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.877536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.875737 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7409e4f-0b82-4884-ac1e-56a382f181bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.969487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.938816 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:52.969487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.940089 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phl7s\" (UniqueName: \"kubernetes.io/projected/7b2d4864-c55c-4861-9b7e-24807e02d842-kube-api-access-phl7s\") pod \"payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:52.969487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.967343 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:53.002777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.989904 6727 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 31 12:48:53.002777 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:52.989963 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 31 12:48:53.433953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.433770 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:53.539524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.538623 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs"] Oct 31 12:48:53.552594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.552565 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc"] Oct 31 12:48:53.712891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.712841 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" event={"ID":"b7409e4f-0b82-4884-ac1e-56a382f181bf","Type":"ContainerStarted","Data":"862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07"} Oct 31 12:48:53.713008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.712916 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" event={"ID":"b7409e4f-0b82-4884-ac1e-56a382f181bf","Type":"ContainerStarted","Data":"0a2d689cbd65663e6c11ee7839330037092aad74c461d5f4124f2278325e7c65"} Oct 31 12:48:53.713008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.712947 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:48:53.714486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.714448 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" event={"ID":"7b2d4864-c55c-4861-9b7e-24807e02d842","Type":"ContainerStarted","Data":"bd2792a8dc0bc6063478c5ea64e3de094fc962a3299002cebf7401bc52b7c3c8"} Oct 31 12:48:53.714621 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.714609 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:48:53.739326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.739302 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv"] Oct 31 12:48:53.744825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.744804 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.745737 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.745703 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" podStartSLOduration=1.745691763 podStartE2EDuration="1.745691763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:53.740713445 +0000 UTC m=+1244.555727830" watchObservedRunningTime="2026-10-31 12:48:53.745691763 +0000 UTC m=+1244.560706154" Oct 31 12:48:53.759662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.759640 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv"] Oct 31 12:48:53.819696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.819634 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" podStartSLOduration=1.819615689 podStartE2EDuration="1.819615689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:53.803922554 +0000 UTC m=+1244.618936955" watchObservedRunningTime="2026-10-31 12:48:53.819615689 +0000 UTC m=+1244.634630080" Oct 31 12:48:53.837571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.837544 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.837809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.837796 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.837985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.837973 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.838141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.838119 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.838254 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.838244 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.838435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.838424 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.838884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.838871 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cvp\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-kube-api-access-95cvp\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.839075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.839063 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.839246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.839236 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.942172 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.942134 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.942442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.942419 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.942665 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.942645 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.942861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.942842 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.943200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.943144 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cvp\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-kube-api-access-95cvp\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.944600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.943847 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.944600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.943235 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.944600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.943997 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.944600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.944101 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.944600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.944201 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.944600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.944563 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.945075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.944800 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.945075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.942876 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.945075 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.944886 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.949543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.949484 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.957525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.956562 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.959482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.959445 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:53.960633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.960608 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cvp\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-kube-api-access-95cvp\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:54.003489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:53.980554 6727 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 31 12:48:54.004068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.004028 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 31 12:48:54.069807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.069769 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:54.198664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.198021 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:54.431630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.431600 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv"] Oct 31 12:48:54.433026 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:48:54.432982 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7edc1d_99bc_4df6_93b2_bc945072d2ed.slice/crio-a0b6f983432e9d2b7b13375530be7a96336571c655b0f3ac758f33302fe60b08 WatchSource:0}: Error finding container a0b6f983432e9d2b7b13375530be7a96336571c655b0f3ac758f33302fe60b08: Status 404 returned error can't find the container with id a0b6f983432e9d2b7b13375530be7a96336571c655b0f3ac758f33302fe60b08 Oct 31 12:48:54.436568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.436535 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:54.436652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.436606 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:54.436652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.436635 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:54.441087 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.441050 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:54.682214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.676051 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:54.741779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.741732 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" event={"ID":"7b2d4864-c55c-4861-9b7e-24807e02d842","Type":"ContainerStarted","Data":"29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230"} Oct 31 12:48:54.746552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.746519 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" event={"ID":"2b7edc1d-99bc-4df6-93b2-bc945072d2ed","Type":"ContainerStarted","Data":"3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28"} Oct 31 12:48:54.746716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.746575 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" event={"ID":"2b7edc1d-99bc-4df6-93b2-bc945072d2ed","Type":"ContainerStarted","Data":"a0b6f983432e9d2b7b13375530be7a96336571c655b0f3ac758f33302fe60b08"} Oct 31 12:48:54.782489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.779551 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podStartSLOduration=1.779511426 podStartE2EDuration="1.779511426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:54.777143196 +0000 UTC m=+1245.592157751" watchObservedRunningTime="2026-10-31 12:48:54.779511426 +0000 UTC m=+1245.594525818" Oct 31 12:48:54.974541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.970789 6727 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 31 12:48:54.974541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:54.970847 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 31 12:48:55.087383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:55.087348 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 12:48:55.087967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:55.087940 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:48:55.088074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:55.087940 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 12:48:55.534928 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:55.531882 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:56.055014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:56.054980 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:48:56.056338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:56.056305 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:48:56.057313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:56.057220 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:48:56.093550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:56.089401 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 12:48:56.093550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:56.089457 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 12:48:56.384486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:56.383700 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:56.536108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:56.536075 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:57.087611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.087576 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 12:48:57.087884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.087852 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 12:48:57.097825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.097791 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7"] Oct 31 12:48:57.098861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.098818 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" podUID="433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" containerName="maas-api" containerID="cri-o://4ca962b657715081a3960bb12529287a4fe54435799507512f896ab53bc6c755" gracePeriod=30 Oct 31 12:48:57.195507 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.194995 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:57.372540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.369141 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:57.621396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.615034 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:57.794245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.793078 6727 generic.go:356] "Generic (PLEG): container finished" podID="433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" containerID="4ca962b657715081a3960bb12529287a4fe54435799507512f896ab53bc6c755" exitCode=0 Oct 31 12:48:57.794245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.794181 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" event={"ID":"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c","Type":"ContainerDied","Data":"4ca962b657715081a3960bb12529287a4fe54435799507512f896ab53bc6c755"} Oct 31 12:48:57.852581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.852546 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:57.868540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.867043 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:48:57.950076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.949912 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px"] Oct 31 12:48:57.957049 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.951370 6727 cpu_manager.go:414] "containerMap: removing container" podUID="433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" containerName="maas-api" Oct 31 12:48:57.957049 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.951396 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" containerName="maas-api" Oct 31 12:48:57.957049 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.951745 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" containerName="maas-api" Oct 31 12:48:57.957356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.957235 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:57.964490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:57.964436 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px"] Oct 31 12:48:58.012149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.012087 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:58.014767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.012748 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-metrics-certs\") pod \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " Oct 31 12:48:58.014767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.012928 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-kube-api-access-8s4zd\" (UniqueName: \"kubernetes.io/projected/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-kube-api-access-8s4zd\") pod \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " Oct 31 12:48:58.014767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.013337 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-openshift-service-ca\") pod \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " Oct 31 12:48:58.014767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.013408 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls\") pod \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\" (UID: \"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c\") " Oct 31 12:48:58.023149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.023112 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-openshift-service-ca" pod "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" (UID: "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:58.029725 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.029674 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-kube-api-access-8s4zd" pod "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" (UID: "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c"). InnerVolumeSpecName "kube-api-access-8s4zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:58.030244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.030222 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-metrics-certs" pod "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" (UID: "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:58.077485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.063583 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls" pod "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" (UID: "433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:58.105586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.095820 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.118845 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.118001 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119041 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119111 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119188 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r767n\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-kube-api-access-r767n\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119235 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119355 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119508 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119537 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119647 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119948 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119968 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4zd\" (UniqueName: \"kubernetes.io/projected/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-kube-api-access-8s4zd\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119983 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:58.120117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.119998 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:58.221004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.220924 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221233 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221435 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221581 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r767n\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-kube-api-access-r767n\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221351 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221693 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221695 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221780 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221883 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.221954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.221908 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.222035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.222010 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.222166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.222142 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.222558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.222536 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.222810 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.222763 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.224262 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.224235 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.224353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.224313 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.237070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.234000 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.239008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.238983 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r767n\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-kube-api-access-r767n\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.446691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.446662 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:48:58.806289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.806201 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" Oct 31 12:48:58.806289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.806223 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" event={"ID":"433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c","Type":"ContainerDied","Data":"4630957d612eb9bdddbc4a7ca98edd72e8452da1dbc41fee269acde359afb36c"} Oct 31 12:48:58.806555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.806294 6727 scope.go:122] "RemoveContainer" containerID="4ca962b657715081a3960bb12529287a4fe54435799507512f896ab53bc6c755" Oct 31 12:48:58.833114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.833045 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px"] Oct 31 12:48:58.844207 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.842713 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:58.848435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.848388 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:58.848571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.848501 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:58.848571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.848547 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7"] Oct 31 12:48:58.848654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.848549 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:48:58.857703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:58.854628 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7"] Oct 31 12:48:59.092482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.082861 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 12:48:59.092482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.082933 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 12:48:59.313924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.313895 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" path="/var/lib/kubelet/pods/433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c/volumes" Oct 31 12:48:59.693857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.693821 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:59.850487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.839054 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:59.905522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.898846 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" event={"ID":"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3","Type":"ContainerStarted","Data":"912bfd35c0aac2f815276b50eaec7247a99d9fd99b9280062315a3e6b94bca58"} Oct 31 12:48:59.905522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.898916 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" event={"ID":"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3","Type":"ContainerStarted","Data":"3d30067f7623ea2fe8d2f2dcb3f91fd799961293bf5971da6f8ca5141d5c9aaf"} Oct 31 12:48:59.938117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:48:59.938055 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podStartSLOduration=6.938034593 podStartE2EDuration="6.938034593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:59.927954126 +0000 UTC m=+1250.742968519" watchObservedRunningTime="2026-10-31 12:48:59.938034593 +0000 UTC m=+1250.753048985" Oct 31 12:49:00.086497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:00.085863 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 12:49:00.086497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:00.085922 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 12:49:00.463539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:00.454321 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:49:00.463539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:00.455816 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 12:49:00.463539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:00.455865 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 12:49:00.687550 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:49:00.679685 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:49:00.849754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:00.849713 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5815-7bdf9bf658-wdzc7" podUID="433dd5d2-0296-45fb-bd5e-8b4b9b0bc25c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.116:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:49:01.084826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:01.084791 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 12:49:01.085101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:01.085072 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 12:49:01.117904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:01.117815 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:01.282116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:01.282082 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:01.447070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:01.446986 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 12:49:01.447070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:01.447043 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 12:49:02.058723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:02.058688 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:02.135490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:02.129087 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:49:02.135490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:02.130190 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:49:02.153636 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:02.153603 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:49:02.496482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:02.452755 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 12:49:02.558932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:02.558884 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 12:49:03.122915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:03.122882 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:03.448013 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:03.447943 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 12:49:03.448183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:03.448006 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 12:49:04.116998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:04.116966 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:04.447997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:04.447916 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 12:49:04.448394 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:04.448010 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 12:49:04.750270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:04.750189 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:49:04.915721 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:04.915683 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:05.060439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:05.060359 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:05.321852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:05.321812 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:05.447951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:05.447917 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 12:49:05.448121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:05.448003 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 12:49:05.725374 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:05.725301 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:05.762803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:05.762772 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:49:06.518525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:06.511633 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:49:06.519311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:06.519288 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:49:06.539483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:06.525725 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:49:06.636289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:06.636255 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:06.851014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:06.850960 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:07.008221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:07.008185 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:07.169965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:07.169872 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:07.681072 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:07.681035 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:08.130850 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:08.130814 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:08.836352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:08.836314 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:09.505201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:09.505162 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:10.138406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:10.138375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:11.015187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:11.015150 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:11.187315 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:11.187272 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:11.847941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:11.847896 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:12.149554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:12.147586 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:13.349305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:13.349267 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:14.134294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:14.133966 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:14.866709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:14.866672 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:15.755223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:15.755181 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:16.324715 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:16.324665 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:16.625850 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:16.625774 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:16.784765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:16.784734 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:17.400487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:17.397448 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:17.580290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:17.580252 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9"] Oct 31 12:49:17.721341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:17.721219 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:17.874100 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:17.874064 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:18.033588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:18.033519 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:18.307544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:18.306750 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:19.071382 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:19.071332 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:19.853255 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:19.853174 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:20.538134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:20.535372 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:21.329904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:21.329871 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:22.060846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.060813 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:22.581103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.581060 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" podUID="c4858fc3-28c5-468e-95ec-04009bcb9b32" containerName="payload-pre-processing" containerID="cri-o://ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990" gracePeriod=25 Oct 31 12:49:22.831575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.830728 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:22.936327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.936301 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:49:22.991204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.991169 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4858fc3-28c5-468e-95ec-04009bcb9b32-kube-api-access-rlzp2\" (UniqueName: \"kubernetes.io/projected/c4858fc3-28c5-468e-95ec-04009bcb9b32-kube-api-access-rlzp2\") pod \"c4858fc3-28c5-468e-95ec-04009bcb9b32\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " Oct 31 12:49:22.991401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.991260 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4858fc3-28c5-468e-95ec-04009bcb9b32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4858fc3-28c5-468e-95ec-04009bcb9b32-plugins-config-volume\") pod \"c4858fc3-28c5-468e-95ec-04009bcb9b32\" (UID: \"c4858fc3-28c5-468e-95ec-04009bcb9b32\") " Oct 31 12:49:22.992617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.992154 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4858fc3-28c5-468e-95ec-04009bcb9b32-plugins-config-volume" pod "c4858fc3-28c5-468e-95ec-04009bcb9b32" (UID: "c4858fc3-28c5-468e-95ec-04009bcb9b32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:49:22.992617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.992490 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4858fc3-28c5-468e-95ec-04009bcb9b32-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:22.994761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:22.994734 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4858fc3-28c5-468e-95ec-04009bcb9b32-kube-api-access-rlzp2" pod "c4858fc3-28c5-468e-95ec-04009bcb9b32" (UID: "c4858fc3-28c5-468e-95ec-04009bcb9b32"). InnerVolumeSpecName "kube-api-access-rlzp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:23.094615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.094540 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzp2\" (UniqueName: \"kubernetes.io/projected/c4858fc3-28c5-468e-95ec-04009bcb9b32-kube-api-access-rlzp2\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:23.384997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.384924 6727 generic.go:356] "Generic (PLEG): container finished" podID="c4858fc3-28c5-468e-95ec-04009bcb9b32" containerID="ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990" exitCode=0 Oct 31 12:49:23.384997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.384986 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" Oct 31 12:49:23.385202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.385004 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" event={"ID":"c4858fc3-28c5-468e-95ec-04009bcb9b32","Type":"ContainerDied","Data":"ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990"} Oct 31 12:49:23.385202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.385058 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9" event={"ID":"c4858fc3-28c5-468e-95ec-04009bcb9b32","Type":"ContainerDied","Data":"952b9af884a81675cbae74cbd7f8222d539dd90c4370bbedbcb684edc9113c0c"} Oct 31 12:49:23.385202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.385084 6727 scope.go:122] "RemoveContainer" containerID="ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990" Oct 31 12:49:23.395972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.395955 6727 scope.go:122] "RemoveContainer" containerID="ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990" Oct 31 12:49:23.396246 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:49:23.396221 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990\": container with ID starting with ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990 not found: ID does not exist" containerID="ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990" Oct 31 12:49:23.396305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.396248 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990"} err="failed to get container status \"ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990\": rpc error: code = NotFound desc = could not find container \"ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990\": container with ID starting with ae13ba2c1caf95a3864041824a1c14934a07a00a90e8c474d5c06c92cf14c990 not found: ID does not exist" Oct 31 12:49:23.408327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.408303 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9"] Oct 31 12:49:23.415037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.415005 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-787316-67bd7d8575-fxxg9"] Oct 31 12:49:23.650889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:23.650821 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:24.676950 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:24.676914 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:25.069571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:25.069546 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:25.308154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:25.308118 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4858fc3-28c5-468e-95ec-04009bcb9b32" path="/var/lib/kubelet/pods/c4858fc3-28c5-468e-95ec-04009bcb9b32/volumes" Oct 31 12:49:25.537001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:25.536848 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:26.220972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:26.220937 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:26.398527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:26.398491 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:26.937932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:26.937838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:27.640670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:27.639891 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:28.471174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:28.471126 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:29.117137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:29.117083 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:29.873745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:29.873706 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:30.347517 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:49:30.347442 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:49:30.542037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:30.541991 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:31.415290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:31.415188 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:32.038014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:32.037985 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:49:32.159769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:32.159708 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:32.573233 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:32.573205 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:33.046106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:33.045998 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:34.155045 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:34.155011 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:35.102336 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:35.102303 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:35.241863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:35.241814 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:35.696076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:35.696037 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:36.263484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:36.238120 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:37.449114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:37.449078 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:37.930722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:37.930686 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:38.414532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:38.414498 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:38.595095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:38.595020 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:38.881678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:38.881646 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94"] Oct 31 12:49:38.881879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:38.881850 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" containerID="cri-o://a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960" gracePeriod=30 Oct 31 12:49:39.031695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:39.031662 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:39.789221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:39.789188 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:39.948788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:39.948746 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:40.742262 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:40.742226 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:41.534851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:41.534815 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:42.486129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:42.486093 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:43.249140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:43.249106 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:44.035965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.035932 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:44.271029 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.271001 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:49:44.342369 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.342329 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q"] Oct 31 12:49:44.343769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.343747 6727 cpu_manager.go:414] "containerMap: removing container" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" Oct 31 12:49:44.343860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.343771 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" Oct 31 12:49:44.343912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.343899 6727 cpu_manager.go:414] "containerMap: removing container" podUID="c4858fc3-28c5-468e-95ec-04009bcb9b32" containerName="payload-pre-processing" Oct 31 12:49:44.343963 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.343911 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4858fc3-28c5-468e-95ec-04009bcb9b32" containerName="payload-pre-processing" Oct 31 12:49:44.344142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.344126 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4858fc3-28c5-468e-95ec-04009bcb9b32" containerName="payload-pre-processing" Oct 31 12:49:44.344179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.344152 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerName="istio-proxy" Oct 31 12:49:44.348966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.348945 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.364493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.360283 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1a7d0e-openshift-default-dockercfg-qkjx5\"" Oct 31 12:49:44.369351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.369329 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q"] Oct 31 12:49:44.380220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380191 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-podinfo\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.380347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380258 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-socket\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.380347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380316 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:44.380347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380325 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-envoy\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.380526 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380509 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-kube-api-access-czmmp\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-kube-api-access-czmmp\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.380572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380559 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-credential-socket\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.380702 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380683 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istiod-ca-cert\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.380935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380904 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-certs\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.381033 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380972 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-token\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-token\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.381088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.381064 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-data\") pod \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\" (UID: \"6f85dc17-8edb-4c31-92a7-99afba1dc8c6\") " Oct 31 12:49:44.381544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.380757 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-socket" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:44.382524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.382505 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.384002 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.381288 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-credential-socket" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:44.384156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.381521 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-certs" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:44.384156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.383086 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istiod-ca-cert" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:49:44.384156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.383324 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-data" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:44.384683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.384659 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-podinfo" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:49:44.386601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.386574 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-token" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:44.386693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.386601 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-envoy" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:44.387491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.387438 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-kube-api-access-czmmp" pod "6f85dc17-8edb-4c31-92a7-99afba1dc8c6" (UID: "6f85dc17-8edb-4c31-92a7-99afba1dc8c6"). InnerVolumeSpecName "kube-api-access-czmmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:44.486195 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.486086 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-credential-socket\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.486378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.486331 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-socket\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.486438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.486392 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-token\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.486612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.486588 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57f48ec-1e88-4fb4-9569-7443b23107fa-istiod-ca-cert\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.486693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.486633 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-data\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.486840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.486814 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-certs\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.487054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487005 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlx9\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-kube-api-access-vwlx9\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.487191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487074 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-envoy\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.487191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487139 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-podinfo\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.487363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487344 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.487428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487368 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.487428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487384 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czmmp\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-kube-api-access-czmmp\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.487428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487401 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.487428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487417 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.487720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487432 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.487720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487447 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.487720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.487511 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f85dc17-8edb-4c31-92a7-99afba1dc8c6-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:44.575836 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.575801 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:44.588919 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.588884 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-socket\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.588933 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-token\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.588998 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57f48ec-1e88-4fb4-9569-7443b23107fa-istiod-ca-cert\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589019 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-data\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589096 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-certs\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589249 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlx9\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-kube-api-access-vwlx9\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589311 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-envoy\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589339 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-podinfo\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589424 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-credential-socket\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589489 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-data\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589521 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-certs\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589638 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-socket\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589866 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-credential-socket\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.589948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.589893 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57f48ec-1e88-4fb4-9569-7443b23107fa-istiod-ca-cert\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.591579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.591559 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-envoy\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.591811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.591793 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-podinfo\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.603364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.603340 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlx9\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-kube-api-access-vwlx9\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.603458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.603404 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-token\") pod \"e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.673518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.673021 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:44.705195 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.699436 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks"] Oct 31 12:49:44.705195 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.700057 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" containerID="cri-o://0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d" gracePeriod=30 Oct 31 12:49:44.767561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.762029 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:44.767561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.762962 6727 generic.go:356] "Generic (PLEG): container finished" podID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" containerID="a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960" exitCode=0 Oct 31 12:49:44.767561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.763041 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" event={"ID":"6f85dc17-8edb-4c31-92a7-99afba1dc8c6","Type":"ContainerDied","Data":"a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960"} Oct 31 12:49:44.767561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.763106 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" event={"ID":"6f85dc17-8edb-4c31-92a7-99afba1dc8c6","Type":"ContainerDied","Data":"a78f3e4f42613310cab44ab70b617a7d6bc052138c7698bb4d67452437522bef"} Oct 31 12:49:44.767561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.763140 6727 scope.go:122] "RemoveContainer" containerID="a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960" Oct 31 12:49:44.767561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:44.764905 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94" Oct 31 12:49:45.010501 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.009406 6727 scope.go:122] "RemoveContainer" containerID="a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960" Oct 31 12:49:45.027535 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:49:45.020379 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960\": container with ID starting with a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960 not found: ID does not exist" containerID="a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960" Oct 31 12:49:45.027535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.020420 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960"} err="failed to get container status \"a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960\": rpc error: code = NotFound desc = could not find container \"a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960\": container with ID starting with a0c393c54c9c1cbcf795702359754e602be0c3300cf4b336bbb550b3e9867960 not found: ID does not exist" Oct 31 12:49:45.088109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.069716 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94"] Oct 31 12:49:45.088109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.079651 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-c7m94"] Oct 31 12:49:45.111442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.111301 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:45.199712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.199109 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q"] Oct 31 12:49:45.201929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.201887 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:45.202053 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.201981 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:45.202053 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.202035 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:45.321531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.321496 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f85dc17-8edb-4c31-92a7-99afba1dc8c6" path="/var/lib/kubelet/pods/6f85dc17-8edb-4c31-92a7-99afba1dc8c6/volumes" Oct 31 12:49:45.775154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.775116 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" event={"ID":"f57f48ec-1e88-4fb4-9569-7443b23107fa","Type":"ContainerStarted","Data":"916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e"} Oct 31 12:49:45.775435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.775183 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" event={"ID":"f57f48ec-1e88-4fb4-9569-7443b23107fa","Type":"ContainerStarted","Data":"bcbe95fd18fc99072e629b8b659887bd519c0662f3694714acfb97235200c995"} Oct 31 12:49:45.800104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.800054 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" podStartSLOduration=28.800040187 podStartE2EDuration="28.800040187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:45.798343812 +0000 UTC m=+1296.613358198" watchObservedRunningTime="2026-10-31 12:49:45.800040187 +0000 UTC m=+1296.615054577" Oct 31 12:49:45.845305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:45.845274 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:46.586698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:46.586653 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:46.673373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:46.673339 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:46.691275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:46.691232 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:46.785476 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:46.785435 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:46.786283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:46.786260 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:49:47.712973 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:47.712032 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:48.245290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:48.245258 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:48.490095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:48.490065 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:48.869025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:48.868987 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:49.511459 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:49.511427 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:49.760986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:49.759370 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:50.183144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.183112 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:49:50.239248 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.239213 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l"] Oct 31 12:49:50.240637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.240610 6727 cpu_manager.go:414] "containerMap: removing container" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" Oct 31 12:49:50.240637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.240634 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" Oct 31 12:49:50.240842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.240829 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerName="istio-proxy" Oct 31 12:49:50.244603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.244579 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-certs\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.244751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.244732 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-data\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.244877 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.244861 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-kube-api-access-kpfxb\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-kube-api-access-kpfxb\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.244944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.244880 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-certs" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:50.245001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.244947 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-podinfo\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.245001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.244994 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-envoy\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.245116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.245055 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-data" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:50.245116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.245074 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-socket\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.245220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.245133 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istiod-ca-cert\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.246693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.245266 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-token\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-token\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.246693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.245318 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-credential-socket\") pod \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\" (UID: \"fe1e9ab1-14f7-487e-90c8-b208d18436e6\") " Oct 31 12:49:50.246693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.246119 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istiod-ca-cert" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:49:50.246918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.246872 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.246918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.246889 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.246918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.246898 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.247196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.247052 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-credential-socket" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:50.247334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.247291 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-socket" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:50.249763 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.249742 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.260767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.252662 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:50.260767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.257893 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a0ea4-openshift-default-dockercfg-lllmm\"" Oct 31 12:49:50.270099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.270069 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l"] Oct 31 12:49:50.285413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.285364 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-token" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:50.285710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.285672 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-envoy" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:50.286117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.286092 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-podinfo" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:49:50.286652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.286628 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-kube-api-access-kpfxb" pod "fe1e9ab1-14f7-487e-90c8-b208d18436e6" (UID: "fe1e9ab1-14f7-487e-90c8-b208d18436e6"). InnerVolumeSpecName "kube-api-access-kpfxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:50.350631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.350577 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-data\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.350899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.350641 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-certs\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.350899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.350671 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-credential-socket\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.350899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.350802 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bf9\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-kube-api-access-m7bf9\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.351126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.350926 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istiod-ca-cert\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.351126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351062 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-envoy\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.351231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351191 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-socket\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.351323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351306 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-podinfo\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.351397 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351335 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-token\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.351510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351496 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpfxb\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-kube-api-access-kpfxb\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.351586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351515 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.351586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351529 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.351586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351542 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.351586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351556 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1e9ab1-14f7-487e-90c8-b208d18436e6-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.351586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.351572 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1e9ab1-14f7-487e-90c8-b208d18436e6-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:50.452771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.452679 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-data\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.452771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.452737 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-certs\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.452771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.452764 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-credential-socket\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.452827 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bf9\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-kube-api-access-m7bf9\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.452904 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istiod-ca-cert\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453015 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-envoy\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453112 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-data\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453139 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-socket\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453149 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-certs\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453250 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-credential-socket\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453277 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-podinfo\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453301 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-token\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453409 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-socket\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.453717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.453694 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istiod-ca-cert\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.455762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.455744 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-envoy\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.456096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.456075 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-podinfo\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.467260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.467231 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bf9\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-kube-api-access-m7bf9\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.467597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.467575 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-token\") pod \"e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.622409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.622380 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:50.913606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.913568 6727 generic.go:356] "Generic (PLEG): container finished" podID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" containerID="0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d" exitCode=0 Oct 31 12:49:50.913807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.913649 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" Oct 31 12:49:50.913807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.913662 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" event={"ID":"fe1e9ab1-14f7-487e-90c8-b208d18436e6","Type":"ContainerDied","Data":"0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d"} Oct 31 12:49:50.913807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.913763 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks" event={"ID":"fe1e9ab1-14f7-487e-90c8-b208d18436e6","Type":"ContainerDied","Data":"4f12d564bd3c72c1df1799e49d3878ae98b987a915ce2328f578b32a342bf521"} Oct 31 12:49:50.913807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.913801 6727 scope.go:122] "RemoveContainer" containerID="0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d" Oct 31 12:49:50.939888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.939863 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l"] Oct 31 12:49:50.941191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.941175 6727 scope.go:122] "RemoveContainer" containerID="0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d" Oct 31 12:49:50.941891 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:49:50.941869 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d\": container with ID starting with 0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d not found: ID does not exist" containerID="0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d" Oct 31 12:49:50.941979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.941902 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d"} err="failed to get container status \"0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d\": rpc error: code = NotFound desc = could not find container \"0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d\": container with ID starting with 0a7bc1d04de8ae78335b85d11e2c0eae8759e33e3fa3787edd070c4522508d0d not found: ID does not exist" Oct 31 12:49:50.947625 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:49:50.947594 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c34587a_5bfb_42a1_bad1_cc897fd2fc3b.slice/crio-a054f59b21af8fd0dce903f49dce3340409fc18e07756c98afe0ee9e4ff053ef WatchSource:0}: Error finding container a054f59b21af8fd0dce903f49dce3340409fc18e07756c98afe0ee9e4ff053ef: Status 404 returned error can't find the container with id a054f59b21af8fd0dce903f49dce3340409fc18e07756c98afe0ee9e4ff053ef Oct 31 12:49:50.950122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.950091 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:50.950225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.950170 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:50.950331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.950235 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:50.963072 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.961046 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks"] Oct 31 12:49:50.965638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:50.965615 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-b5mks"] Oct 31 12:49:51.309157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:51.309082 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1e9ab1-14f7-487e-90c8-b208d18436e6" path="/var/lib/kubelet/pods/fe1e9ab1-14f7-487e-90c8-b208d18436e6/volumes" Oct 31 12:49:51.374213 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:51.371700 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:51.925366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:51.925330 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" event={"ID":"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b","Type":"ContainerStarted","Data":"9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69"} Oct 31 12:49:51.925581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:51.925398 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" event={"ID":"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b","Type":"ContainerStarted","Data":"a054f59b21af8fd0dce903f49dce3340409fc18e07756c98afe0ee9e4ff053ef"} Oct 31 12:49:51.960371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:51.960306 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" podStartSLOduration=13.960289665 podStartE2EDuration="13.960289665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:51.955761127 +0000 UTC m=+1302.770775521" watchObservedRunningTime="2026-10-31 12:49:51.960289665 +0000 UTC m=+1302.775304059" Oct 31 12:49:52.241522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:52.241421 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:52.622631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:52.622599 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:52.628675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:52.628647 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:52.934655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:52.934580 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:52.935544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:52.935524 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:49:53.162345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:53.162314 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:53.887352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:53.887316 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn"] Oct 31 12:49:53.887788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:53.887612 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" podUID="4271c7a4-9c32-4b80-b9b0-9615ba75612a" containerName="maas-api" containerID="cri-o://a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984" gracePeriod=30 Oct 31 12:49:53.952700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:53.952658 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:54.144282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.143958 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:54.308729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.293443 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:54.411020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.410958 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:49:54.490834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.490799 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4271c7a4-9c32-4b80-b9b0-9615ba75612a-kube-api-access-5dxd5\" (UniqueName: \"kubernetes.io/projected/4271c7a4-9c32-4b80-b9b0-9615ba75612a-kube-api-access-5dxd5\") pod \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " Oct 31 12:49:54.491234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.491210 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-metrics-certs\") pod \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " Oct 31 12:49:54.491396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.491378 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4271c7a4-9c32-4b80-b9b0-9615ba75612a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271c7a4-9c32-4b80-b9b0-9615ba75612a-openshift-service-ca\") pod \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " Oct 31 12:49:54.491552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.491448 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls\") pod \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\" (UID: \"4271c7a4-9c32-4b80-b9b0-9615ba75612a\") " Oct 31 12:49:54.491821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.491788 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4271c7a4-9c32-4b80-b9b0-9615ba75612a-openshift-service-ca" pod "4271c7a4-9c32-4b80-b9b0-9615ba75612a" (UID: "4271c7a4-9c32-4b80-b9b0-9615ba75612a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:49:54.492432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.492408 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4271c7a4-9c32-4b80-b9b0-9615ba75612a-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:54.493648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.493620 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-metrics-certs" pod "4271c7a4-9c32-4b80-b9b0-9615ba75612a" (UID: "4271c7a4-9c32-4b80-b9b0-9615ba75612a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:49:54.493648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.493634 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls" pod "4271c7a4-9c32-4b80-b9b0-9615ba75612a" (UID: "4271c7a4-9c32-4b80-b9b0-9615ba75612a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:49:54.493803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.493663 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4271c7a4-9c32-4b80-b9b0-9615ba75612a-kube-api-access-5dxd5" pod "4271c7a4-9c32-4b80-b9b0-9615ba75612a" (UID: "4271c7a4-9c32-4b80-b9b0-9615ba75612a"). InnerVolumeSpecName "kube-api-access-5dxd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:54.510872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.510844 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7"] Oct 31 12:49:54.511811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.511782 6727 cpu_manager.go:414] "containerMap: removing container" podUID="4271c7a4-9c32-4b80-b9b0-9615ba75612a" containerName="maas-api" Oct 31 12:49:54.511811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.511800 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4271c7a4-9c32-4b80-b9b0-9615ba75612a" containerName="maas-api" Oct 31 12:49:54.512432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.512017 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4271c7a4-9c32-4b80-b9b0-9615ba75612a" containerName="maas-api" Oct 31 12:49:54.517047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.517024 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.538837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.536106 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4e8eb9-openshift-default-dockercfg-sczrg\"" Oct 31 12:49:54.547544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.547516 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7"] Oct 31 12:49:54.594497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.594442 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560131f7-efd5-4343-8378-6efcb60cc69c-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.594693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.594522 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.594693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.594676 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrr24\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-kube-api-access-xrr24\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.594819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.594762 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.594875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.594854 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.594936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.594891 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.595098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.595079 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560131f7-efd5-4343-8378-6efcb60cc69c-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.595197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.595114 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.595296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.595281 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.595419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.595405 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:54.595494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.595425 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4271c7a4-9c32-4b80-b9b0-9615ba75612a-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:54.595494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.595441 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxd5\" (UniqueName: \"kubernetes.io/projected/4271c7a4-9c32-4b80-b9b0-9615ba75612a-kube-api-access-5dxd5\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:54.697202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697126 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560131f7-efd5-4343-8378-6efcb60cc69c-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697165 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697301 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697352 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560131f7-efd5-4343-8378-6efcb60cc69c-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697389 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697555 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrr24\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-kube-api-access-xrr24\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697644 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697677 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697731 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697743 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697815 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.697987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697966 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560131f7-efd5-4343-8378-6efcb60cc69c-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.698039 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.697970 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.698215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.698191 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.699846 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.699828 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.699983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.699964 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560131f7-efd5-4343-8378-6efcb60cc69c-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.710408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.710385 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.726412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.726388 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrr24\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-kube-api-access-xrr24\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.805473 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.805424 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:54.859806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.859770 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:54.969058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.968962 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" Oct 31 12:49:54.970574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.969017 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" event={"ID":"4271c7a4-9c32-4b80-b9b0-9615ba75612a","Type":"ContainerDied","Data":"a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984"} Oct 31 12:49:54.970574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.968955 6727 generic.go:356] "Generic (PLEG): container finished" podID="4271c7a4-9c32-4b80-b9b0-9615ba75612a" containerID="a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984" exitCode=0 Oct 31 12:49:54.970574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.970577 6727 scope.go:122] "RemoveContainer" containerID="a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984" Oct 31 12:49:54.970818 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.970742 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn" event={"ID":"4271c7a4-9c32-4b80-b9b0-9615ba75612a","Type":"ContainerDied","Data":"ed1c83b81fdf39cf863e4df7f1887aa6ace2fd3742ed21bba656fcaf673f7c6f"} Oct 31 12:49:54.991519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.991417 6727 scope.go:122] "RemoveContainer" containerID="a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984" Oct 31 12:49:54.991826 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:49:54.991787 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984\": container with ID starting with a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984 not found: ID does not exist" containerID="a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984" Oct 31 12:49:54.991876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:54.991828 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984"} err="failed to get container status \"a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984\": rpc error: code = NotFound desc = could not find container \"a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984\": container with ID starting with a18ee414a45374fd9566e2b26adfadf73603a80482d8418476f625cbdced2984 not found: ID does not exist" Oct 31 12:49:55.020054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.020025 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn"] Oct 31 12:49:55.032889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.032857 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-787316-586f895476-dcqqn"] Oct 31 12:49:55.048252 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.048151 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:55.237229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.237171 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:55.237486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.237447 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:55.237688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.237661 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:49:55.243482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.242806 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7"] Oct 31 12:49:55.391163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:55.391131 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4271c7a4-9c32-4b80-b9b0-9615ba75612a" path="/var/lib/kubelet/pods/4271c7a4-9c32-4b80-b9b0-9615ba75612a/volumes" Oct 31 12:49:56.001517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.001441 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:56.013807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.013773 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" event={"ID":"560131f7-efd5-4343-8378-6efcb60cc69c","Type":"ContainerStarted","Data":"ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a"} Oct 31 12:49:56.013978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.013829 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" event={"ID":"560131f7-efd5-4343-8378-6efcb60cc69c","Type":"ContainerStarted","Data":"7ccdd5253708a98279364d2148672c3da92fd81199f0e5100dd30e9cdc13cdec"} Oct 31 12:49:56.024993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.024966 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw"] Oct 31 12:49:56.030194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.030175 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.045851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.043134 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2a0ea4-dockercfg-fhtv8\"" Oct 31 12:49:56.045851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.045131 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw"] Oct 31 12:49:56.053184 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.053161 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2a0ea4\"" Oct 31 12:49:56.071651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.071558 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" podStartSLOduration=9.071533135 podStartE2EDuration="9.071533135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:56.057835612 +0000 UTC m=+1306.872850004" watchObservedRunningTime="2026-10-31 12:49:56.071533135 +0000 UTC m=+1306.886547527" Oct 31 12:49:56.214498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.206263 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.214498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.206580 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnqq\" (UniqueName: \"kubernetes.io/projected/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-kube-api-access-xjnqq\") pod \"payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.215736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.215698 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:56.339530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.337564 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnqq\" (UniqueName: \"kubernetes.io/projected/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-kube-api-access-xjnqq\") pod \"payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.339530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.337987 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.339530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.339110 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.422489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.422244 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnqq\" (UniqueName: \"kubernetes.io/projected/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-kube-api-access-xjnqq\") pod \"payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.660754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.660456 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:56.897079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.873185 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:49:56.897079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.873407 6727 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 12:49:56.897079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:56.873475 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 12:49:57.318334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:57.318310 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw"] Oct 31 12:49:57.442527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:57.442488 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:57.860708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:57.860673 6727 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 12:49:57.860918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:57.860734 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 12:49:58.043721 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:58.043688 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" event={"ID":"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f","Type":"ContainerStarted","Data":"8aae6faa1ea836fab6cd83cc9a6e69ccc06b12b9bf25ae305dc7e28d7c027b20"} Oct 31 12:49:58.044201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:58.044171 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:49:58.044393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:58.044254 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" event={"ID":"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f","Type":"ContainerStarted","Data":"738d1f43d4bcc6be644fcefd47e195ee00f53f11d3845e59e19b4ecdb00e2a1c"} Oct 31 12:49:58.097446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:58.097381 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" podStartSLOduration=3.097359635 podStartE2EDuration="3.097359635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:58.094824903 +0000 UTC m=+1308.909839294" watchObservedRunningTime="2026-10-31 12:49:58.097359635 +0000 UTC m=+1308.912374027" Oct 31 12:49:58.299847 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:58.299769 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:58.860726 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:58.860696 6727 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 12:49:58.861140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:58.860745 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 12:49:59.248303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:59.247832 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:59.860444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:59.860413 6727 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 12:49:59.860661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:49:59.860549 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 12:50:00.047407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:00.041345 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:00.436574 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:50:00.435567 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:50:00.860448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:00.860413 6727 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 12:50:00.860636 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:00.860487 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 12:50:01.517727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:01.517690 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:02.036482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:02.034550 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:50:02.036482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:02.035796 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:50:02.052948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:02.052870 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:50:02.834286 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:02.834238 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:03.696923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:03.696886 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:04.685223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:04.685196 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:05.001666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:05.001558 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:05.236104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:05.236072 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:05.415768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:05.415727 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:06.012740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:06.012698 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:06.224705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:06.224667 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:06.681086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:06.681052 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:06.869496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:06.868062 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:07.758379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:07.758340 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:07.938991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:07.938953 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:08.554415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:08.554382 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:08.729370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:08.729329 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:09.037074 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:09.037030 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:09.569147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:09.569108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:10.123971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:10.123942 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:50:10.644871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:10.644816 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:11.486712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:11.486673 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:12.557276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:12.557238 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:13.449188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:13.449151 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:14.177054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:14.177021 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:15.190504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:15.190443 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:15.378929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:15.378889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:15.924077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:15.924009 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:18.503687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:18.503611 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:18.697405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:18.697356 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:18.959761 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:18.959725 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:19.843278 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:19.842239 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:20.585743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:20.585695 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:21.427132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:21.427088 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:22.267154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:22.267049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:23.005404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:23.005376 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:23.837967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:23.837938 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:24.572493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:24.571608 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:25.341488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:25.338539 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:26.344022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:26.343989 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:27.099839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:27.099805 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:28.050237 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:28.050191 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:28.958561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:28.958513 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:29.724619 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:29.724580 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:30.346751 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:50:30.346710 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:50:30.487228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:30.487192 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:50:30.626444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:30.626356 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:31.565226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:31.565194 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:32.774488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:32.773839 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:34.128584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:34.128529 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:35.616234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:35.616157 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:35.896660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:35.896557 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:35.910586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:35.910545 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:35.987592 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:35.987550 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.024829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.024786 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.196683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.196568 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.329487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.317956 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.410193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.330403 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.410193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.331667 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.410193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.331941 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:50:36.410193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.343794 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:50:36.410193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.344065 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 12:50:36.455566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.449090 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 12:50:36.912629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.911945 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.925516 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.913553 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:50:36.940288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.915340 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:36.985488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.970192 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:50:37.306541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:36.988793 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:50:37.306541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.168894 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:37.306541 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.253932 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:50:37.336333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.336292 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:50:37.371079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.370222 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:50:37.371079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.370299 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:50:37.371079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.370386 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:50:37.371079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.370421 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:50:37.371079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.370453 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:50:37.371079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.370507 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:50:37.371079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.370536 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:50:37.414571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.414541 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:50:37.418045 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.418012 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:50:37.426729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.423496 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:50:37.588970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:37.588888 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:38.906802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:38.906775 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:39.847985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:39.847953 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:40.560193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:40.560156 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:41.257278 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:41.257197 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:42.119486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:42.111108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:42.873402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:42.873353 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:43.757226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:43.757135 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:44.033484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:44.033397 6727 generic.go:356] "Generic (PLEG): container finished" podID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerID="916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e" exitCode=0 Oct 31 12:50:44.033626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:44.033475 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" event={"ID":"f57f48ec-1e88-4fb4-9569-7443b23107fa","Type":"ContainerDied","Data":"916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e"} Oct 31 12:50:44.033891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:44.033870 6727 scope.go:122] "RemoveContainer" containerID="916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e" Oct 31 12:50:44.035785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:44.035755 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:50:44.035904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:44.035835 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:50:44.035904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:44.035884 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:50:44.641867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:44.641822 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:45.048600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:45.048488 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" event={"ID":"f57f48ec-1e88-4fb4-9569-7443b23107fa","Type":"ContainerStarted","Data":"f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f"} Oct 31 12:50:45.413616 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:45.413579 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:45.688099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:45.687949 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:50:45.689170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:45.689137 6727 patch_prober.go:29] interesting pod/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 12:50:45.689331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:45.689204 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 12:50:46.312395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:46.308136 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:46.679488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:46.676734 6727 patch_prober.go:29] interesting pod/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 12:50:46.679488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:46.676789 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 12:50:46.785577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:46.785545 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:50:47.093388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:47.093353 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:47.674945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:47.674890 6727 patch_prober.go:29] interesting pod/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 12:50:47.675587 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:47.675541 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 12:50:47.840921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:47.840889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:48.544185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:48.544155 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:48.679904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:48.679861 6727 patch_prober.go:29] interesting pod/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 12:50:48.680556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:48.680518 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 12:50:48.695366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:48.695332 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:49.086498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:49.077019 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:49.254134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:49.254102 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:49.679539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:49.673839 6727 patch_prober.go:29] interesting pod/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 12:50:49.679539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:49.673914 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 12:50:50.053832 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:50.053760 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:50.734138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:50.734105 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:50:50.735949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:50.735927 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:50:51.088054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:51.088003 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:51.874663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:51.874611 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:52.658582 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:52.658548 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:53.565487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:53.564042 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:54.293613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:54.290722 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:55.040646 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:55.040606 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:55.730620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:55.730588 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:56.471403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:56.471361 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:56.868421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:56.868392 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:57.275939 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:57.275844 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:57.466953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:57.466920 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:58.076388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:58.076351 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:58.850038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:58.849990 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:59.710957 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:50:59.710924 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:00.344548 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:51:00.344514 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:51:00.441440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:00.441400 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:01.176012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:01.175970 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:01.896543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:01.896428 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:02.605769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:02.605709 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:03.375970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:03.375933 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:04.137265 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:04.137232 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:04.828258 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:04.828230 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:05.572504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:05.566432 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:06.270431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:06.270395 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:06.927514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:06.927447 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:07.643841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:07.643804 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:08.006025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:08.005942 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:08.426162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:08.426126 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:09.098101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:09.098066 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:09.752123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:09.752080 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:10.358820 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:10.358791 6727 scope.go:122] "RemoveContainer" containerID="fd8a5f296ae2279dc1d02ab83a5fa9a96c9a7e1cf80c0e73506efa5be3485916" Oct 31 12:51:10.442452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:10.442419 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:11.225979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:11.225942 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:11.910034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:11.909993 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:12.710557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:12.710522 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:13.340044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:13.340007 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:14.609775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:14.609745 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:14.841013 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:14.840977 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:15.582626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:15.582543 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:16.326548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:16.326502 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:16.652643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:16.652557 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:51:17.136770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:17.135084 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:18.432976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:18.432928 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:19.885484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:19.883713 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:20.223597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.202988 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 12:51:20.245488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.238539 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 12:51:20.245488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.239599 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.270487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.269551 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.295487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.288656 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.320488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.300549 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.357489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.348542 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.382488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.364714 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.425502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.424904 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 12:51:20.429485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.425826 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.429485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.426099 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 12:51:20.753486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.752690 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:20.903489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:20.902568 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.183489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.169689 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.211487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.210562 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.211487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.211165 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:51:21.260522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.259548 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 12:51:21.260522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.259651 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.260522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.260519 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:51:21.260840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.260581 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:51:21.276487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.272835 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.276487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.273306 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:51:21.298485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.285271 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:21.298485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.292659 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.298485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.293116 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:51:21.323518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.319552 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 12:51:21.323518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.322001 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:51:21.378487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.377967 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 12:51:21.403489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.379108 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.434536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.433567 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:21.510489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.445533 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 12:51:21.952283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.951949 6727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978442 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978528 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978564 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978598 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978630 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978661 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978693 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978724 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978752 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978780 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:51:22.005429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:21.978811 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:51:22.054977 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:22.054941 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:51:22.211537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:22.194877 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 12:51:22.241564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:22.240537 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:51:22.648794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:22.648756 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:23.028439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:23.028355 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:23.186559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:23.186523 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:23.785489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:23.784996 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:24.617564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:24.616047 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:25.558962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:25.558889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:26.246991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:26.246957 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:27.123890 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:27.123851 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:27.754514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:27.754476 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:28.512135 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:28.512097 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:29.267387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.267287 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:29.931414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.931385 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:29.973715 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.973671 6727 generic.go:356] "Generic (PLEG): container finished" podID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerID="9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69" exitCode=0 Oct 31 12:51:29.974111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.974069 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" event={"ID":"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b","Type":"ContainerDied","Data":"9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69"} Oct 31 12:51:29.974816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.974792 6727 scope.go:122] "RemoveContainer" containerID="9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69" Oct 31 12:51:29.982754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.982705 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:51:29.983104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.983069 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:51:29.983290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:29.983272 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:51:30.350684 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:51:30.350645 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:51:30.792454 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:30.792319 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:31.003560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:31.003527 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" event={"ID":"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b","Type":"ContainerStarted","Data":"1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232"} Oct 31 12:51:31.626070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:31.625998 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:31.627283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:31.627248 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:51:31.627566 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:31.627546 6727 patch_prober.go:29] interesting pod/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 12:51:31.627733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:31.627710 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 12:51:32.448975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:32.448737 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:32.622805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:32.622768 6727 patch_prober.go:29] interesting pod/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 12:51:32.622978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:32.622828 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 12:51:33.295150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:33.294950 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:33.623319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:33.623285 6727 patch_prober.go:29] interesting pod/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 12:51:33.623543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:33.623343 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 12:51:34.105488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:34.105365 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:34.623741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:34.623699 6727 patch_prober.go:29] interesting pod/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 12:51:34.624151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:34.623761 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 12:51:34.853050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:34.853018 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:35.622873 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:35.622832 6727 patch_prober.go:29] interesting pod/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 12:51:35.623079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:35.622891 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 12:51:35.780354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:35.780319 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:36.515999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:36.515962 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:36.628932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:36.628899 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:51:36.629314 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:36.629289 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:51:36.631200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:36.631175 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:51:37.284363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:37.284327 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:38.081655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:38.081625 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:38.814630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:38.814595 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:39.530598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:39.530561 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:40.233973 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:40.233930 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:41.006122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:41.005020 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:41.768067 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:41.768032 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:42.434141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:42.434096 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:43.144288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:43.144257 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:43.837111 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:43.837081 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:44.558939 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:44.558906 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:45.433346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:45.433312 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:46.313842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:46.313807 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:47.184601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:47.184140 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:48.016331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:48.016299 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:48.855893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:48.843744 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:49.560348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:49.560321 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:50.377940 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:50.377911 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:51.335782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:51.335745 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:52.060038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:52.059998 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:52.896348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:52.896318 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:53.606903 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:53.606877 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:54.562834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:54.562800 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:55.253430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:55.253397 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:56.216123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:56.216070 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:57.091565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:57.091523 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:58.027228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:58.022958 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:59.083718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:59.083684 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:59.741247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:51:59.741201 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:00.344727 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:52:00.344692 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:52:00.428548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:00.428518 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:01.179281 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:01.179235 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:02.006292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:02.006210 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:02.652349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:02.652317 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:03.473487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:03.453760 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:04.207783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:04.207757 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:04.882642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:04.882605 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:05.599182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:05.599139 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:06.381668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:06.381637 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:07.039931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:07.039899 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:07.732921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:07.732883 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:08.034377 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:08.034304 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:08.451568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:08.451531 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:09.150539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:09.150432 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:09.846195 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:09.846113 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:10.632786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:10.632740 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:10.801303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:10.801272 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:11.468171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:11.467522 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:12.129947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:12.129914 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:13.045553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:13.045521 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:13.770561 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:13.770527 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:14.563059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:14.563021 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:15.213635 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:15.213592 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:15.289027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:15.288995 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:52:16.030539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:16.030413 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:16.783684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:16.783649 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:17.554743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:17.554708 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:18.307008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:18.306975 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:19.083612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:19.083574 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:19.831009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:19.830933 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:20.565700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:20.565657 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:21.228410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:21.228375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:21.980030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:21.979993 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:22.856937 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:22.856909 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:23.466939 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:23.466902 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:24.176839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:24.176809 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:25.041098 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:25.041060 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:25.710487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:25.708977 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:26.517063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:26.517030 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:27.375026 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:27.374995 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:28.131630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:28.131596 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:28.939526 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:28.939483 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:29.572327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:29.572290 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:30.309932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:30.309898 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:30.361981 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:52:30.361940 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:52:31.059689 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:31.059655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:31.962316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:31.962279 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:32.628453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:32.628424 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:33.414419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:33.409125 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:34.078711 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:34.078568 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:35.007170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:35.007134 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:35.717852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:35.717815 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:36.535154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:36.535112 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:37.281435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:37.279606 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:38.180053 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:38.180016 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:39.105661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:39.105626 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:39.869322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:39.869290 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:40.577613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:40.577553 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:41.418234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:41.418198 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:42.127584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:42.127550 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:42.861411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:42.861364 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:43.622008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:43.621969 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:44.338661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:44.338519 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:45.057430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:45.057394 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:45.742863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:45.741515 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:46.508011 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:46.507971 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:47.497386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:47.497356 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:48.316576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:48.316539 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:49.156651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:49.156589 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:49.876231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:49.876193 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:50.669532 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:50.666986 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:51.536156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:51.536123 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:52.326157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:52.324986 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:53.076110 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:53.076074 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:53.902321 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:53.899958 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:54.194151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:54.194048 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k"] Oct 31 12:52:54.194334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:54.194309 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" containerID="cri-o://64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794" gracePeriod=30 Oct 31 12:52:54.311092 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:54.311055 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml"] Oct 31 12:52:54.311307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:54.311285 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" containerID="cri-o://de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4" gracePeriod=30 Oct 31 12:52:54.839654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:54.839617 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:55.641239 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:55.641189 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:56.369593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:56.366961 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:57.151090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.151049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:57.417844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.417760 6727 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:52:57.417844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.417829 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:52:57.444227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.444188 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw"] Oct 31 12:52:57.467616 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.467586 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475"] Oct 31 12:52:57.562244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.562208 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:57.720061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.719954 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:57.884499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:57.877365 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:58.487690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:58.487656 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:58.675970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:58.675938 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:58.857162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:58.857132 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:59.034863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.034837 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:59.439956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.439857 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:59.532714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.532681 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:52:59.582493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.573872 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8"] Oct 31 12:52:59.582493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.575294 6727 cpu_manager.go:414] "containerMap: removing container" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" Oct 31 12:52:59.582493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.575315 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" Oct 31 12:52:59.582493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.575671 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" Oct 31 12:52:59.582493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.575690 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" Oct 31 12:52:59.582493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.580290 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.595322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.592071 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d176a5-openshift-default-dockercfg-nrx6h\"" Oct 31 12:52:59.595322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.593986 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8"] Oct 31 12:52:59.625486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625439 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-istio-token\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-istio-token\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.625619 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625521 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-data\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.625619 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625546 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-kube-api-access-tg8md\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-kube-api-access-tg8md\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.625750 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625652 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-envoy\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.625750 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625734 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868f4085-7faa-463d-8ac2-66b99c188160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f4085-7faa-463d-8ac2-66b99c188160-istiod-ca-cert\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.625854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625825 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-data" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.625910 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625858 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-certs\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.625967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.625906 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-credential-socket\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.626452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626045 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/868f4085-7faa-463d-8ac2-66b99c188160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f4085-7faa-463d-8ac2-66b99c188160-istio-podinfo\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.626452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626122 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-socket\") pod \"868f4085-7faa-463d-8ac2-66b99c188160\" (UID: \"868f4085-7faa-463d-8ac2-66b99c188160\") " Oct 31 12:52:59.626452 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626455 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-data\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.626729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626518 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-credential-socket\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.626729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626554 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-socket\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.626729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626587 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istiod-ca-cert\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.626729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626638 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfjzs\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-kube-api-access-wfjzs\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.626992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626812 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-envoy\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.626992 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.626863 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-credential-socket" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.627099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627049 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-socket" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.627099 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627085 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-certs" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.627355 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627334 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-token\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.627524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627400 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-certs\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.627524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627476 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-podinfo\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.627824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627808 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.627901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627829 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.627901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627844 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.627901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.627859 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.628064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.628021 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868f4085-7faa-463d-8ac2-66b99c188160-istiod-ca-cert" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:59.629564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.629531 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/868f4085-7faa-463d-8ac2-66b99c188160-istio-podinfo" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:52:59.631043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.631015 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-envoy" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.631866 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.631842 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-istio-token" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:59.633018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.632993 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-kube-api-access-tg8md" pod "868f4085-7faa-463d-8ac2-66b99c188160" (UID: "868f4085-7faa-463d-8ac2-66b99c188160"). InnerVolumeSpecName "kube-api-access-tg8md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:59.640807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.640788 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:52:59.697134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.697099 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn"] Oct 31 12:52:59.698329 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.698304 6727 cpu_manager.go:414] "containerMap: removing container" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" Oct 31 12:52:59.698329 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.698324 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" Oct 31 12:52:59.698554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.698358 6727 cpu_manager.go:414] "containerMap: removing container" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" Oct 31 12:52:59.698554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.698363 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868f4085-7faa-463d-8ac2-66b99c188160" containerName="istio-proxy" Oct 31 12:52:59.698554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.698542 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerName="istio-proxy" Oct 31 12:52:59.703848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.703824 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.710834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.709325 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn"] Oct 31 12:52:59.710834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.710657 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9b62a0-openshift-default-dockercfg-l2m24\"" Oct 31 12:52:59.729694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.729662 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istiod-ca-cert\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.729852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.729804 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-kube-api-access-qkcwt\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-kube-api-access-qkcwt\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.729922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.729875 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-certs\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.729980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.729919 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-podinfo\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.730212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730191 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-token\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-token\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.730308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730292 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-credential-socket\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.730372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730328 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-socket\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.730514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730380 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-envoy\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.730514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730425 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-data\") pod \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\" (UID: \"25e7214c-ed96-4ac6-a2b4-e7e954ce2673\") " Oct 31 12:52:59.730514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730427 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-certs" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.730716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730691 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-credential-socket" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.730808 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-podinfo\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731057 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-data\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731101 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-credential-socket\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731135 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-socket\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731168 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istiod-ca-cert\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731218 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfjzs\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-kube-api-access-wfjzs\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731288 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-data" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731387 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-envoy\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731571 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-token\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731634 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-certs\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731796 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731815 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731830 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8md\" (UniqueName: \"kubernetes.io/projected/868f4085-7faa-463d-8ac2-66b99c188160-kube-api-access-tg8md\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731844 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f4085-7faa-463d-8ac2-66b99c188160-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731856 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731870 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f4085-7faa-463d-8ac2-66b99c188160-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731884 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f4085-7faa-463d-8ac2-66b99c188160-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.731896 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.732670 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-certs\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.732869 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-socket" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.733202 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istiod-ca-cert" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:59.733555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.733567 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-credential-socket\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.734603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.733815 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-data\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.734603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.734076 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-socket\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.734779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.734758 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istiod-ca-cert\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.742539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.742480 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-envoy" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:59.742953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.742894 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-podinfo\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.743170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.743127 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-kube-api-access-qkcwt" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "kube-api-access-qkcwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:59.743824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.743796 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-podinfo" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:52:59.744173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.744149 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-token" pod "25e7214c-ed96-4ac6-a2b4-e7e954ce2673" (UID: "25e7214c-ed96-4ac6-a2b4-e7e954ce2673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:59.744751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.744729 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfjzs\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-kube-api-access-wfjzs\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.745779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.745755 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-envoy\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.746216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.746196 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-token\") pod \"e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.833319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.833287 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-token\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.833540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.833516 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-envoy\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.833620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.833580 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzlj\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-kube-api-access-ttzlj\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.833796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.833779 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-podinfo\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.833879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.833807 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-credential-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.833879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.833843 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istiod-ca-cert\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.834007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.833982 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-certs\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.834148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834121 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.834284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834173 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-data\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.834513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834447 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.834513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834510 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.834660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834527 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.834660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834545 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.834660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834559 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkcwt\" (UniqueName: \"kubernetes.io/projected/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-kube-api-access-qkcwt\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.834660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.834573 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e7214c-ed96-4ac6-a2b4-e7e954ce2673-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:59.904405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.904369 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.943736 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-podinfo\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.943796 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-credential-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.943850 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istiod-ca-cert\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.943919 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-certs\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.944022 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.944073 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-data\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.944156 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-token\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.944330 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-envoy\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.946327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.944368 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzlj\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-kube-api-access-ttzlj\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.953572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.948760 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:59.957674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.956042 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-credential-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.957674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.955897 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-data\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.957674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.956610 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-certs\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.957674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.957114 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istiod-ca-cert\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.957674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.957588 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.959988 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.959948 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-envoy\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.960850 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.960824 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-token\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.963140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.962854 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-podinfo\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:52:59.965163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:52:59.965141 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzlj\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-kube-api-access-ttzlj\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:53:00.030990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.030843 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:53:00.066836 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.066799 6727 generic.go:356] "Generic (PLEG): container finished" podID="868f4085-7faa-463d-8ac2-66b99c188160" containerID="64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794" exitCode=0 Oct 31 12:53:00.066975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.066902 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" event={"ID":"868f4085-7faa-463d-8ac2-66b99c188160","Type":"ContainerDied","Data":"64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794"} Oct 31 12:53:00.066975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.066968 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" event={"ID":"868f4085-7faa-463d-8ac2-66b99c188160","Type":"ContainerDied","Data":"6329912f0ad7d603684fc732682b2943b384d750973cc4d391d6cc5572932049"} Oct 31 12:53:00.067106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.066999 6727 scope.go:122] "RemoveContainer" containerID="64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794" Oct 31 12:53:00.067248 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.067231 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k" Oct 31 12:53:00.077612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.077577 6727 generic.go:356] "Generic (PLEG): container finished" podID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" containerID="de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4" exitCode=0 Oct 31 12:53:00.077748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.077625 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" event={"ID":"25e7214c-ed96-4ac6-a2b4-e7e954ce2673","Type":"ContainerDied","Data":"de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4"} Oct 31 12:53:00.077748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.077670 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" event={"ID":"25e7214c-ed96-4ac6-a2b4-e7e954ce2673","Type":"ContainerDied","Data":"c6533498b5e77583653e85c6a40797b9bf8e7f59ddf3069ee9439675998f7a38"} Oct 31 12:53:00.077748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.077746 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml" Oct 31 12:53:00.114488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.114387 6727 scope.go:122] "RemoveContainer" containerID="b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db" Oct 31 12:53:00.142742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.142710 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k"] Oct 31 12:53:00.146211 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.143280 6727 scope.go:122] "RemoveContainer" containerID="64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794" Oct 31 12:53:00.146720 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:00.146676 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794\": container with ID starting with 64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794 not found: ID does not exist" containerID="64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794" Oct 31 12:53:00.146831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.146748 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794"} err="failed to get container status \"64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794\": rpc error: code = NotFound desc = could not find container \"64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794\": container with ID starting with 64576869e374b296193306d4855d56481a3814f6f04d750a53c494631620f794 not found: ID does not exist" Oct 31 12:53:00.146831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.146776 6727 scope.go:122] "RemoveContainer" containerID="b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db" Oct 31 12:53:00.147070 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:00.147049 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db\": container with ID starting with b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db not found: ID does not exist" containerID="b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db" Oct 31 12:53:00.147144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.147077 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db"} err="failed to get container status \"b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db\": rpc error: code = NotFound desc = could not find container \"b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db\": container with ID starting with b5afcf9c57e5bd8e30ef87eb8b370a64ce01c5f60ffee4a93335cf8fe1af08db not found: ID does not exist" Oct 31 12:53:00.147144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.147092 6727 scope.go:122] "RemoveContainer" containerID="de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4" Oct 31 12:53:00.153088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.153059 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-bw46k"] Oct 31 12:53:00.162968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.162940 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml"] Oct 31 12:53:00.170070 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.170032 6727 scope.go:122] "RemoveContainer" containerID="de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4" Oct 31 12:53:00.170511 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:00.170439 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4\": container with ID starting with de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4 not found: ID does not exist" containerID="de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4" Oct 31 12:53:00.170511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.170489 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4"} err="failed to get container status \"de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4\": rpc error: code = NotFound desc = could not find container \"de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4\": container with ID starting with de423ec3618a95552286e0a641ecc796ad3c21f4d793a064a6bc09606c2b39f4 not found: ID does not exist" Oct 31 12:53:00.170649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.170531 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wb2ml"] Oct 31 12:53:00.269360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.269329 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8"] Oct 31 12:53:00.273288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.273246 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:53:00.273404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.273350 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:53:00.273404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.273394 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:53:00.361700 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:00.361667 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:53:00.400677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.398665 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn"] Oct 31 12:53:00.400677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.400141 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:53:00.400677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.400233 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:53:00.401037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.401005 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:53:00.992309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:00.992248 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:01.098096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.098037 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" event={"ID":"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa","Type":"ContainerStarted","Data":"baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a"} Oct 31 12:53:01.098290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.098130 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" event={"ID":"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa","Type":"ContainerStarted","Data":"2259b46b083fa4af1ab109d30742397fe26a920b9c51c460260dbf33595cbee2"} Oct 31 12:53:01.136017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.135960 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podStartSLOduration=124.135939683 podStartE2EDuration="2m4.135939683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:01.130621783 +0000 UTC m=+1491.945636177" watchObservedRunningTime="2026-10-31 12:53:01.135939683 +0000 UTC m=+1491.950954073" Oct 31 12:53:01.139445 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.139409 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" event={"ID":"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d","Type":"ContainerStarted","Data":"475c06dc335837dbd5890eed6c40fd6c7a69a2c25d26216fe9103d845df196f3"} Oct 31 12:53:01.139641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.139485 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" event={"ID":"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d","Type":"ContainerStarted","Data":"9b67a99e878f58621212a8f1b37dc68ae3021b244339f80659cc14862b5e668e"} Oct 31 12:53:01.188194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.188142 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podStartSLOduration=133.188128205 podStartE2EDuration="2m13.188128205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:01.184824182 +0000 UTC m=+1491.999838572" watchObservedRunningTime="2026-10-31 12:53:01.188128205 +0000 UTC m=+1492.003142595" Oct 31 12:53:01.305421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.305346 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e7214c-ed96-4ac6-a2b4-e7e954ce2673" path="/var/lib/kubelet/pods/25e7214c-ed96-4ac6-a2b4-e7e954ce2673/volumes" Oct 31 12:53:01.305911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.305895 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868f4085-7faa-463d-8ac2-66b99c188160" path="/var/lib/kubelet/pods/868f4085-7faa-463d-8ac2-66b99c188160/volumes" Oct 31 12:53:01.905543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.905510 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:53:01.919806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:01.919777 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:53:02.034841 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.034810 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:53:02.040799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.040772 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:53:02.145176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.145105 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:02.162202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.159141 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:53:02.162202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.159198 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:53:02.162202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.159520 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:53:02.164100 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.164076 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:53:02.444607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.444453 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" podUID="573a12f7-4680-46bc-83f4-d65080d436ad" containerName="payload-processing" containerID="cri-o://d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4" gracePeriod=25 Oct 31 12:53:02.468917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.468887 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" podUID="69719755-33c7-41bd-9ec1-1feadbf10a47" containerName="payload-pre-processing" containerID="cri-o://4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c" gracePeriod=25 Oct 31 12:53:02.814963 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.814933 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:53:02.869437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.869407 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:53:02.889239 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.889214 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69719755-33c7-41bd-9ec1-1feadbf10a47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69719755-33c7-41bd-9ec1-1feadbf10a47-plugins-config-volume\") pod \"69719755-33c7-41bd-9ec1-1feadbf10a47\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " Oct 31 12:53:02.889405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.889293 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573a12f7-4680-46bc-83f4-d65080d436ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573a12f7-4680-46bc-83f4-d65080d436ad-plugins-config-volume\") pod \"573a12f7-4680-46bc-83f4-d65080d436ad\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " Oct 31 12:53:02.889610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.889591 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573a12f7-4680-46bc-83f4-d65080d436ad-kube-api-access-hr5h5\" (UniqueName: \"kubernetes.io/projected/573a12f7-4680-46bc-83f4-d65080d436ad-kube-api-access-hr5h5\") pod \"573a12f7-4680-46bc-83f4-d65080d436ad\" (UID: \"573a12f7-4680-46bc-83f4-d65080d436ad\") " Oct 31 12:53:02.889710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.889608 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69719755-33c7-41bd-9ec1-1feadbf10a47-plugins-config-volume" pod "69719755-33c7-41bd-9ec1-1feadbf10a47" (UID: "69719755-33c7-41bd-9ec1-1feadbf10a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:02.889710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.889690 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69719755-33c7-41bd-9ec1-1feadbf10a47-kube-api-access-572h4\" (UniqueName: \"kubernetes.io/projected/69719755-33c7-41bd-9ec1-1feadbf10a47-kube-api-access-572h4\") pod \"69719755-33c7-41bd-9ec1-1feadbf10a47\" (UID: \"69719755-33c7-41bd-9ec1-1feadbf10a47\") " Oct 31 12:53:02.889825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.889720 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573a12f7-4680-46bc-83f4-d65080d436ad-plugins-config-volume" pod "573a12f7-4680-46bc-83f4-d65080d436ad" (UID: "573a12f7-4680-46bc-83f4-d65080d436ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:02.890774 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.890749 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69719755-33c7-41bd-9ec1-1feadbf10a47-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:02.890884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.890778 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573a12f7-4680-46bc-83f4-d65080d436ad-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:02.891900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.891876 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69719755-33c7-41bd-9ec1-1feadbf10a47-kube-api-access-572h4" pod "69719755-33c7-41bd-9ec1-1feadbf10a47" (UID: "69719755-33c7-41bd-9ec1-1feadbf10a47"). InnerVolumeSpecName "kube-api-access-572h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:02.891986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.891904 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573a12f7-4680-46bc-83f4-d65080d436ad-kube-api-access-hr5h5" pod "573a12f7-4680-46bc-83f4-d65080d436ad" (UID: "573a12f7-4680-46bc-83f4-d65080d436ad"). InnerVolumeSpecName "kube-api-access-hr5h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:02.992611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.992580 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5h5\" (UniqueName: \"kubernetes.io/projected/573a12f7-4680-46bc-83f4-d65080d436ad-kube-api-access-hr5h5\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:02.992611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:02.992606 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572h4\" (UniqueName: \"kubernetes.io/projected/69719755-33c7-41bd-9ec1-1feadbf10a47-kube-api-access-572h4\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:03.167904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.167872 6727 generic.go:356] "Generic (PLEG): container finished" podID="573a12f7-4680-46bc-83f4-d65080d436ad" containerID="d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4" exitCode=0 Oct 31 12:53:03.168390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.167932 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" Oct 31 12:53:03.168390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.167968 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" event={"ID":"573a12f7-4680-46bc-83f4-d65080d436ad","Type":"ContainerDied","Data":"d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4"} Oct 31 12:53:03.168390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.168020 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw" event={"ID":"573a12f7-4680-46bc-83f4-d65080d436ad","Type":"ContainerDied","Data":"3a1e215f6c749ae457b931420fe1af74b06afd1a58bc3bd8ca49c90c738feb22"} Oct 31 12:53:03.168390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.168043 6727 scope.go:122] "RemoveContainer" containerID="d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4" Oct 31 12:53:03.169605 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.169585 6727 generic.go:356] "Generic (PLEG): container finished" podID="69719755-33c7-41bd-9ec1-1feadbf10a47" containerID="4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c" exitCode=0 Oct 31 12:53:03.169769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.169642 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" Oct 31 12:53:03.170068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.170024 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" event={"ID":"69719755-33c7-41bd-9ec1-1feadbf10a47","Type":"ContainerDied","Data":"4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c"} Oct 31 12:53:03.170173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.170112 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475" event={"ID":"69719755-33c7-41bd-9ec1-1feadbf10a47","Type":"ContainerDied","Data":"805b3cc17b0072df0299982f8b2b184ef69e6c0c487af0283257ad09154fe377"} Oct 31 12:53:03.181025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.181006 6727 scope.go:122] "RemoveContainer" containerID="d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4" Oct 31 12:53:03.181350 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:03.181331 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4\": container with ID starting with d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4 not found: ID does not exist" containerID="d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4" Oct 31 12:53:03.181417 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.181359 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4"} err="failed to get container status \"d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4\": rpc error: code = NotFound desc = could not find container \"d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4\": container with ID starting with d4ea388a436c9b3312619a29f27a34c1ef5150c680da735b73083fc0e815c3c4 not found: ID does not exist" Oct 31 12:53:03.181417 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.181375 6727 scope.go:122] "RemoveContainer" containerID="4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c" Oct 31 12:53:03.193282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.193261 6727 scope.go:122] "RemoveContainer" containerID="4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c" Oct 31 12:53:03.193564 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:03.193544 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c\": container with ID starting with 4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c not found: ID does not exist" containerID="4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c" Oct 31 12:53:03.193652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.193571 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c"} err="failed to get container status \"4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c\": rpc error: code = NotFound desc = could not find container \"4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c\": container with ID starting with 4682ed2d99a5ee837bfa025ac037f2d96b9f1c1566dd5822f35beb327949ca4c not found: ID does not exist" Oct 31 12:53:03.201923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.201900 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475"] Oct 31 12:53:03.210011 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.209988 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e43d8b-6f675f6c7f-g5475"] Oct 31 12:53:03.222402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.222375 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw"] Oct 31 12:53:03.226372 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.226353 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e43d8b-77c789ffd9-bhzsw"] Oct 31 12:53:03.304525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.304497 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573a12f7-4680-46bc-83f4-d65080d436ad" path="/var/lib/kubelet/pods/573a12f7-4680-46bc-83f4-d65080d436ad/volumes" Oct 31 12:53:03.304948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.304932 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69719755-33c7-41bd-9ec1-1feadbf10a47" path="/var/lib/kubelet/pods/69719755-33c7-41bd-9ec1-1feadbf10a47/volumes" Oct 31 12:53:03.516488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.514542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:03.747140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.747005 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:03.902752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:03.902716 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:04.090806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:04.090774 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:04.282734 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:04.282697 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:04.720235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:04.720195 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:06.023730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.023699 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:06.777256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.777223 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:06.848480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.848437 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79"] Oct 31 12:53:06.849398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.849376 6727 cpu_manager.go:414] "containerMap: removing container" podUID="69719755-33c7-41bd-9ec1-1feadbf10a47" containerName="payload-pre-processing" Oct 31 12:53:06.849398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.849395 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69719755-33c7-41bd-9ec1-1feadbf10a47" containerName="payload-pre-processing" Oct 31 12:53:06.849585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.849438 6727 cpu_manager.go:414] "containerMap: removing container" podUID="573a12f7-4680-46bc-83f4-d65080d436ad" containerName="payload-processing" Oct 31 12:53:06.849585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.849444 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573a12f7-4680-46bc-83f4-d65080d436ad" containerName="payload-processing" Oct 31 12:53:06.849585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.849585 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="573a12f7-4680-46bc-83f4-d65080d436ad" containerName="payload-processing" Oct 31 12:53:06.849676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.849598 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="69719755-33c7-41bd-9ec1-1feadbf10a47" containerName="payload-pre-processing" Oct 31 12:53:06.855741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.855722 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:06.873503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.870402 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9b62a0\"" Oct 31 12:53:06.873503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.870648 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9b62a0-dockercfg-9szkp\"" Oct 31 12:53:06.873503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.872957 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79"] Oct 31 12:53:06.906800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.906770 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5"] Oct 31 12:53:06.912035 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.912004 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5a92fb-09f8-4caf-8a18-652b94bfa460-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:06.912323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.912301 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxlf\" (UniqueName: \"kubernetes.io/projected/3c5a92fb-09f8-4caf-8a18-652b94bfa460-kube-api-access-8bxlf\") pod \"payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:06.915546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.915520 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:06.925433 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:06.924574 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5"] Oct 31 12:53:07.014627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.014598 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5a92fb-09f8-4caf-8a18-652b94bfa460-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:07.014792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.014752 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:07.014792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.014783 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxlf\" (UniqueName: \"kubernetes.io/projected/3c5a92fb-09f8-4caf-8a18-652b94bfa460-kube-api-access-8bxlf\") pod \"payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:07.014888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.014818 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxb6b\" (UniqueName: \"kubernetes.io/projected/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-kube-api-access-nxb6b\") pod \"payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:07.015333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.015309 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5a92fb-09f8-4caf-8a18-652b94bfa460-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:07.038028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.037952 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxlf\" (UniqueName: \"kubernetes.io/projected/3c5a92fb-09f8-4caf-8a18-652b94bfa460-kube-api-access-8bxlf\") pod \"payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:07.100888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.091148 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:07.126487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.120451 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:07.126487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.120581 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxb6b\" (UniqueName: \"kubernetes.io/projected/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-kube-api-access-nxb6b\") pod \"payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:07.172487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.155305 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:07.172487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.160650 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxb6b\" (UniqueName: \"kubernetes.io/projected/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-kube-api-access-nxb6b\") pod \"payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:07.181488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.174841 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:07.248487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.245699 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:07.290801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.290729 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:07.797577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.797514 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5"] Oct 31 12:53:07.802763 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:07.802252 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79"] Oct 31 12:53:08.311845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.298391 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" event={"ID":"3c5a92fb-09f8-4caf-8a18-652b94bfa460","Type":"ContainerStarted","Data":"7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3"} Oct 31 12:53:08.311845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.298481 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" event={"ID":"3c5a92fb-09f8-4caf-8a18-652b94bfa460","Type":"ContainerStarted","Data":"b85ef06c9df04dac142027d6f09e9d3d0a0ed478d34b71849ca6d74fe4ba66eb"} Oct 31 12:53:08.328538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.328508 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:08.381272 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.381177 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:08.383335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.383302 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" event={"ID":"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa","Type":"ContainerStarted","Data":"7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2"} Oct 31 12:53:08.383506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.383364 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" event={"ID":"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa","Type":"ContainerStarted","Data":"05aa8e70a410a2dbab9edc61349728dab0506a6a7bac8d0310513d8bbf881b18"} Oct 31 12:53:08.384308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.384289 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:08.414166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.414075 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" podStartSLOduration=2.414035286 podStartE2EDuration="2.414035286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:08.41054167 +0000 UTC m=+1499.225556062" watchObservedRunningTime="2026-10-31 12:53:08.414035286 +0000 UTC m=+1499.229049708" Oct 31 12:53:08.467893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.467836 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" podStartSLOduration=2.467817845 podStartE2EDuration="2.467817845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:08.459572387 +0000 UTC m=+1499.274586774" watchObservedRunningTime="2026-10-31 12:53:08.467817845 +0000 UTC m=+1499.282832235" Oct 31 12:53:08.711489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.710509 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:09.005548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:08.988780 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:09.352898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:09.352865 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:09.823406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:09.823368 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988"] Oct 31 12:53:09.823676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:09.823642 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" containerID="cri-o://edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20" gracePeriod=30 Oct 31 12:53:10.335837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:10.335802 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:10.497183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:10.497158 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:10.996332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:10.996241 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:12.175924 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:12.175877 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:12.331661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:12.326968 6727 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:53:12.331661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:12.327037 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:53:12.439020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:12.436333 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:12.758090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:12.757988 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:13.005915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:13.005878 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:13.286562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:13.286453 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:13.985354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:13.985319 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:15.015105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.015056 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:15.180527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.180422 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:15.237966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.237936 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350195 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-podinfo\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350399 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-token\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-token\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350523 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-socket\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350596 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-certs\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350686 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istiod-ca-cert\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350813 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-envoy\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350882 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-credential-socket\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.350926 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-kube-api-access-xph7n\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-kube-api-access-xph7n\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.351578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.351104 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-data\") pod \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\" (UID: \"717c9dd9-85aa-4ea6-8510-a40ef7e2af57\") " Oct 31 12:53:15.456705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.398676 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istiod-ca-cert" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:15.456705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.399170 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-certs" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:15.456705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.413683 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-socket" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:15.456705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.429765 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-credential-socket" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:15.545685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.459026 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.545685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.459062 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.545685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.459078 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.545685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.459094 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.545685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.460093 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-data" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:15.605006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.563480 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.707839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.612527 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-token" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:15.708103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.683428 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-envoy" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:15.708214 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.683533 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-kube-api-access-xph7n" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "kube-api-access-xph7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:15.708290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.707644 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-podinfo" pod "717c9dd9-85aa-4ea6-8510-a40ef7e2af57" (UID: "717c9dd9-85aa-4ea6-8510-a40ef7e2af57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:53:15.708360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.704239 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:15.712913 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.712887 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg"] Oct 31 12:53:15.714315 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.714292 6727 cpu_manager.go:414] "containerMap: removing container" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" Oct 31 12:53:15.714315 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.714314 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" Oct 31 12:53:15.714556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.714330 6727 cpu_manager.go:414] "containerMap: removing container" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" Oct 31 12:53:15.714556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.714338 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" Oct 31 12:53:15.714645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.714615 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" Oct 31 12:53:15.714645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.714630 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerName="istio-proxy" Oct 31 12:53:15.716019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.715999 6727 generic.go:356] "Generic (PLEG): container finished" podID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" containerID="edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20" exitCode=0 Oct 31 12:53:15.716275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.716264 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" Oct 31 12:53:15.719542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.719510 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" event={"ID":"717c9dd9-85aa-4ea6-8510-a40ef7e2af57","Type":"ContainerDied","Data":"edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20"} Oct 31 12:53:15.719645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.719550 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg"] Oct 31 12:53:15.719645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.719571 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988" event={"ID":"717c9dd9-85aa-4ea6-8510-a40ef7e2af57","Type":"ContainerDied","Data":"e5deef60ac4d67f8b30dd0884e2d3722b1186c2fe7154fb43d7aed3e3e22c40f"} Oct 31 12:53:15.719645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.719592 6727 scope.go:122] "RemoveContainer" containerID="edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20" Oct 31 12:53:15.719795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.719721 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:15.736534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.736518 6727 scope.go:122] "RemoveContainer" containerID="02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db" Oct 31 12:53:15.745520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.745499 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d086f3\"" Oct 31 12:53:15.747448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.747420 6727 scope.go:122] "RemoveContainer" containerID="edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20" Oct 31 12:53:15.747709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.747691 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d086f3-dockercfg-rv558\"" Oct 31 12:53:15.747779 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:15.747761 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20\": container with ID starting with edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20 not found: ID does not exist" containerID="edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20" Oct 31 12:53:15.747816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.747787 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20"} err="failed to get container status \"edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20\": rpc error: code = NotFound desc = could not find container \"edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20\": container with ID starting with edd492fd878da3ec574dfde6d27628ed465a171c1ead8bca092d126144968c20 not found: ID does not exist" Oct 31 12:53:15.747816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.747804 6727 scope.go:122] "RemoveContainer" containerID="02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db" Oct 31 12:53:15.748038 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:15.748024 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db\": container with ID starting with 02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db not found: ID does not exist" containerID="02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db" Oct 31 12:53:15.748080 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.748041 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db"} err="failed to get container status \"02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db\": rpc error: code = NotFound desc = could not find container \"02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db\": container with ID starting with 02df18b3e7dd2e5ed5f76a7669b64ab38b8b7dad94aac9fd086ea5d5d02c19db not found: ID does not exist" Oct 31 12:53:15.777655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.777625 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988"] Oct 31 12:53:15.801197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.801172 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-8h988"] Oct 31 12:53:15.809918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.809886 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twzbm\" (UniqueName: \"kubernetes.io/projected/dd401d12-5957-4561-93ec-71b6a56e614b-kube-api-access-twzbm\") pod \"payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:15.810084 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.810046 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd401d12-5957-4561-93ec-71b6a56e614b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:15.810222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.810210 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.810264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.810226 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xph7n\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-kube-api-access-xph7n\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.810264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.810236 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.810264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.810247 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717c9dd9-85aa-4ea6-8510-a40ef7e2af57-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:15.912908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.912867 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twzbm\" (UniqueName: \"kubernetes.io/projected/dd401d12-5957-4561-93ec-71b6a56e614b-kube-api-access-twzbm\") pod \"payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:15.913101 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.913049 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd401d12-5957-4561-93ec-71b6a56e614b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:15.914367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.914340 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd401d12-5957-4561-93ec-71b6a56e614b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:15.930411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:15.930389 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twzbm\" (UniqueName: \"kubernetes.io/projected/dd401d12-5957-4561-93ec-71b6a56e614b-kube-api-access-twzbm\") pod \"payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:16.043409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:16.043318 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:16.349702 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:53:16.349664 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd401d12_5957_4561_93ec_71b6a56e614b.slice/crio-39f28b35e4210ca4c4e33020d336a64960331e1d970fab9c0230e63bbf2fc337 WatchSource:0}: Error finding container 39f28b35e4210ca4c4e33020d336a64960331e1d970fab9c0230e63bbf2fc337: Status 404 returned error can't find the container with id 39f28b35e4210ca4c4e33020d336a64960331e1d970fab9c0230e63bbf2fc337 Oct 31 12:53:16.349852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:16.349834 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg"] Oct 31 12:53:16.390218 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:16.390184 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:16.727874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:16.727798 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" event={"ID":"dd401d12-5957-4561-93ec-71b6a56e614b","Type":"ContainerStarted","Data":"d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14"} Oct 31 12:53:16.727874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:16.727847 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" event={"ID":"dd401d12-5957-4561-93ec-71b6a56e614b","Type":"ContainerStarted","Data":"39f28b35e4210ca4c4e33020d336a64960331e1d970fab9c0230e63bbf2fc337"} Oct 31 12:53:16.728069 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:16.728014 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:16.756283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:16.756221 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" podStartSLOduration=1.756205806 podStartE2EDuration="1.756205806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:16.751008128 +0000 UTC m=+1507.566022520" watchObservedRunningTime="2026-10-31 12:53:16.756205806 +0000 UTC m=+1507.571220220" Oct 31 12:53:17.304042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:17.304004 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717c9dd9-85aa-4ea6-8510-a40ef7e2af57" path="/var/lib/kubelet/pods/717c9dd9-85aa-4ea6-8510-a40ef7e2af57/volumes" Oct 31 12:53:17.503546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:17.503507 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:18.847680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:18.847649 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:19.728898 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:19.728860 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:20.465498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:20.465452 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:53:20.465982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:20.465558 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:53:20.678327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:20.678282 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:21.482012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:21.481984 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:21.684162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:21.684127 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:21.923356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:21.923320 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7"] Oct 31 12:53:21.923601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:21.923575 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" containerID="cri-o://ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a" gracePeriod=30 Oct 31 12:53:22.080809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:22.080760 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:23.368056 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:23.368011 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:24.495882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:24.495837 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:25.383264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:25.383226 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:26.534812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:26.517087 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:27.247366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.247339 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:53:27.275488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.272185 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:27.380245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.378475 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/560131f7-efd5-4343-8378-6efcb60cc69c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560131f7-efd5-4343-8378-6efcb60cc69c-istiod-ca-cert\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.379771 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/560131f7-efd5-4343-8378-6efcb60cc69c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560131f7-efd5-4343-8378-6efcb60cc69c-istio-podinfo\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.379814 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-kube-api-access-xrr24\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-kube-api-access-xrr24\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.379908 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-envoy\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.379926 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-socket\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.379990 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-credential-socket\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.380174 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-certs\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.380333 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-data\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.380962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.380398 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-istio-token\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-istio-token\") pod \"560131f7-efd5-4343-8378-6efcb60cc69c\" (UID: \"560131f7-efd5-4343-8378-6efcb60cc69c\") " Oct 31 12:53:27.382772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.382419 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560131f7-efd5-4343-8378-6efcb60cc69c-istiod-ca-cert" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:27.382987 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.382957 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-data" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:27.383199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.383174 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-socket" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:27.383331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.383242 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-certs" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:27.383401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.383375 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-credential-socket" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:27.386239 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.386214 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-istio-token" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:27.387730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.387701 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/560131f7-efd5-4343-8378-6efcb60cc69c-istio-podinfo" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:53:27.387820 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.387770 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-kube-api-access-xrr24" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "kube-api-access-xrr24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:27.393724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.393691 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-envoy" pod "560131f7-efd5-4343-8378-6efcb60cc69c" (UID: "560131f7-efd5-4343-8378-6efcb60cc69c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:27.445576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.445542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483043 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483070 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483080 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483090 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483100 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560131f7-efd5-4343-8378-6efcb60cc69c-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483108 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483117 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560131f7-efd5-4343-8378-6efcb60cc69c-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483130 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560131f7-efd5-4343-8378-6efcb60cc69c-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.483118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.483144 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrr24\" (UniqueName: \"kubernetes.io/projected/560131f7-efd5-4343-8378-6efcb60cc69c-kube-api-access-xrr24\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:27.595198 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.595155 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:27.776723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.776633 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:53:27.918486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.912371 6727 generic.go:356] "Generic (PLEG): container finished" podID="560131f7-efd5-4343-8378-6efcb60cc69c" containerID="ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a" exitCode=0 Oct 31 12:53:27.918486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.912787 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" event={"ID":"560131f7-efd5-4343-8378-6efcb60cc69c","Type":"ContainerDied","Data":"ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a"} Oct 31 12:53:27.918486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.912843 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" event={"ID":"560131f7-efd5-4343-8378-6efcb60cc69c","Type":"ContainerDied","Data":"7ccdd5253708a98279364d2148672c3da92fd81199f0e5100dd30e9cdc13cdec"} Oct 31 12:53:27.918486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.912894 6727 scope.go:122] "RemoveContainer" containerID="ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a" Oct 31 12:53:27.918486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.913168 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7" Oct 31 12:53:27.953010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.952982 6727 scope.go:122] "RemoveContainer" containerID="ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a" Oct 31 12:53:27.953479 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:27.953432 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a\": container with ID starting with ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a not found: ID does not exist" containerID="ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a" Oct 31 12:53:27.953622 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.953549 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a"} err="failed to get container status \"ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a\": rpc error: code = NotFound desc = could not find container \"ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a\": container with ID starting with ba7be0cde64a807889bc94b229fea13470342f4721344d035c58aa8383c3f56a not found: ID does not exist" Oct 31 12:53:27.971600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.971569 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7"] Oct 31 12:53:27.977062 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:27.977040 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-gjvm7"] Oct 31 12:53:28.256797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:28.256754 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:28.412586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:28.411758 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:29.305799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:29.305762 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" path="/var/lib/kubelet/pods/560131f7-efd5-4343-8378-6efcb60cc69c/volumes" Oct 31 12:53:29.383997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:29.383582 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:30.048523 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:30.044538 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:30.380262 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:30.380194 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:53:30.992357 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:30.992329 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:31.789226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:31.789194 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:32.677853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:32.677808 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:32.848672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:32.848637 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:32.854517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:32.854487 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw"] Oct 31 12:53:32.999600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:32.999514 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:33.864968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:33.864918 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:34.122556 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:34.122438 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs"] Oct 31 12:53:34.127587 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:34.127565 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc"] Oct 31 12:53:34.246724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:34.246692 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:34.419306 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:34.419214 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:34.811628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:34.811549 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:35.004946 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:35.004881 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:35.156300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:35.156264 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:35.484825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:35.484734 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:35.728163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:35.728133 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:36.277715 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:36.277688 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:36.603312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:36.603281 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:37.370399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:37.370349 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:37.855117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:37.855076 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" podUID="3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" containerName="payload-pre-processing" containerID="cri-o://8aae6faa1ea836fab6cd83cc9a6e69ccc06b12b9bf25ae305dc7e28d7c027b20" gracePeriod=25 Oct 31 12:53:38.093725 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.085896 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:38.127872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.127786 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" event={"ID":"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f","Type":"ContainerDied","Data":"8aae6faa1ea836fab6cd83cc9a6e69ccc06b12b9bf25ae305dc7e28d7c027b20"} Oct 31 12:53:38.128560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.128528 6727 generic.go:356] "Generic (PLEG): container finished" podID="3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" containerID="8aae6faa1ea836fab6cd83cc9a6e69ccc06b12b9bf25ae305dc7e28d7c027b20" exitCode=0 Oct 31 12:53:38.207638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.207613 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:53:38.303840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.303812 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-plugins-config-volume\") pod \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " Oct 31 12:53:38.304028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.303940 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-kube-api-access-xjnqq\" (UniqueName: \"kubernetes.io/projected/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-kube-api-access-xjnqq\") pod \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\" (UID: \"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f\") " Oct 31 12:53:38.304166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.304143 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-plugins-config-volume" pod "3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" (UID: "3cd39810-98b2-4aac-9fa9-1d2d56f25c2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:38.304453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.304436 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:38.306212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.306185 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-kube-api-access-xjnqq" pod "3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" (UID: "3cd39810-98b2-4aac-9fa9-1d2d56f25c2f"). InnerVolumeSpecName "kube-api-access-xjnqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:38.406079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.405995 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnqq\" (UniqueName: \"kubernetes.io/projected/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f-kube-api-access-xjnqq\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:38.821255 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:38.819787 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:39.123563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.123448 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" podUID="7b2d4864-c55c-4861-9b7e-24807e02d842" containerName="payload-processing" containerID="cri-o://29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230" gracePeriod=25 Oct 31 12:53:39.129050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.129023 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" podUID="b7409e4f-0b82-4884-ac1e-56a382f181bf" containerName="payload-pre-processing" containerID="cri-o://862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07" gracePeriod=25 Oct 31 12:53:39.139535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.139515 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" Oct 31 12:53:39.140115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.139513 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw" event={"ID":"3cd39810-98b2-4aac-9fa9-1d2d56f25c2f","Type":"ContainerDied","Data":"738d1f43d4bcc6be644fcefd47e195ee00f53f11d3845e59e19b4ecdb00e2a1c"} Oct 31 12:53:39.140216 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.140142 6727 scope.go:122] "RemoveContainer" containerID="8aae6faa1ea836fab6cd83cc9a6e69ccc06b12b9bf25ae305dc7e28d7c027b20" Oct 31 12:53:39.277658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.277633 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw"] Oct 31 12:53:39.283707 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.283677 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a0ea4-68fd79c879-lsrmw"] Oct 31 12:53:39.309143 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.308686 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" path="/var/lib/kubelet/pods/3cd39810-98b2-4aac-9fa9-1d2d56f25c2f/volumes" Oct 31 12:53:39.465061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.465027 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:53:39.496959 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.496937 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:53:39.646653 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.646580 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7409e4f-0b82-4884-ac1e-56a382f181bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7409e4f-0b82-4884-ac1e-56a382f181bf-plugins-config-volume\") pod \"b7409e4f-0b82-4884-ac1e-56a382f181bf\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " Oct 31 12:53:39.646800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.646668 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7409e4f-0b82-4884-ac1e-56a382f181bf-kube-api-access-pz8bt\" (UniqueName: \"kubernetes.io/projected/b7409e4f-0b82-4884-ac1e-56a382f181bf-kube-api-access-pz8bt\") pod \"b7409e4f-0b82-4884-ac1e-56a382f181bf\" (UID: \"b7409e4f-0b82-4884-ac1e-56a382f181bf\") " Oct 31 12:53:39.646800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.646745 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2d4864-c55c-4861-9b7e-24807e02d842-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2d4864-c55c-4861-9b7e-24807e02d842-plugins-config-volume\") pod \"7b2d4864-c55c-4861-9b7e-24807e02d842\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " Oct 31 12:53:39.646800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.646790 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2d4864-c55c-4861-9b7e-24807e02d842-kube-api-access-phl7s\" (UniqueName: \"kubernetes.io/projected/7b2d4864-c55c-4861-9b7e-24807e02d842-kube-api-access-phl7s\") pod \"7b2d4864-c55c-4861-9b7e-24807e02d842\" (UID: \"7b2d4864-c55c-4861-9b7e-24807e02d842\") " Oct 31 12:53:39.646983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.646927 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7409e4f-0b82-4884-ac1e-56a382f181bf-plugins-config-volume" pod "b7409e4f-0b82-4884-ac1e-56a382f181bf" (UID: "b7409e4f-0b82-4884-ac1e-56a382f181bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:39.647176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.647153 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2d4864-c55c-4861-9b7e-24807e02d842-plugins-config-volume" pod "7b2d4864-c55c-4861-9b7e-24807e02d842" (UID: "7b2d4864-c55c-4861-9b7e-24807e02d842"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:39.647553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.647534 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7409e4f-0b82-4884-ac1e-56a382f181bf-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:39.647602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.647556 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2d4864-c55c-4861-9b7e-24807e02d842-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:39.648740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.648720 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7409e4f-0b82-4884-ac1e-56a382f181bf-kube-api-access-pz8bt" pod "b7409e4f-0b82-4884-ac1e-56a382f181bf" (UID: "b7409e4f-0b82-4884-ac1e-56a382f181bf"). InnerVolumeSpecName "kube-api-access-pz8bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:39.648795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.648763 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2d4864-c55c-4861-9b7e-24807e02d842-kube-api-access-phl7s" pod "7b2d4864-c55c-4861-9b7e-24807e02d842" (UID: "7b2d4864-c55c-4861-9b7e-24807e02d842"). InnerVolumeSpecName "kube-api-access-phl7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:39.685851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.685814 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:39.753487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.749630 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz8bt\" (UniqueName: \"kubernetes.io/projected/b7409e4f-0b82-4884-ac1e-56a382f181bf-kube-api-access-pz8bt\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:39.753487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:39.749670 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phl7s\" (UniqueName: \"kubernetes.io/projected/7b2d4864-c55c-4861-9b7e-24807e02d842-kube-api-access-phl7s\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:40.149852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.149821 6727 generic.go:356] "Generic (PLEG): container finished" podID="b7409e4f-0b82-4884-ac1e-56a382f181bf" containerID="862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07" exitCode=0 Oct 31 12:53:40.149998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.149868 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" Oct 31 12:53:40.149998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.149897 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" event={"ID":"b7409e4f-0b82-4884-ac1e-56a382f181bf","Type":"ContainerDied","Data":"862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07"} Oct 31 12:53:40.149998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.149982 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc" event={"ID":"b7409e4f-0b82-4884-ac1e-56a382f181bf","Type":"ContainerDied","Data":"0a2d689cbd65663e6c11ee7839330037092aad74c461d5f4124f2278325e7c65"} Oct 31 12:53:40.150220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.150014 6727 scope.go:122] "RemoveContainer" containerID="862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07" Oct 31 12:53:40.152599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.152576 6727 generic.go:356] "Generic (PLEG): container finished" podID="7b2d4864-c55c-4861-9b7e-24807e02d842" containerID="29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230" exitCode=0 Oct 31 12:53:40.152717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.152605 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" event={"ID":"7b2d4864-c55c-4861-9b7e-24807e02d842","Type":"ContainerDied","Data":"29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230"} Oct 31 12:53:40.152717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.152624 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" Oct 31 12:53:40.152717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.152657 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs" event={"ID":"7b2d4864-c55c-4861-9b7e-24807e02d842","Type":"ContainerDied","Data":"bd2792a8dc0bc6063478c5ea64e3de094fc962a3299002cebf7401bc52b7c3c8"} Oct 31 12:53:40.162519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.162496 6727 scope.go:122] "RemoveContainer" containerID="862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07" Oct 31 12:53:40.162793 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:40.162773 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07\": container with ID starting with 862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07 not found: ID does not exist" containerID="862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07" Oct 31 12:53:40.162858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.162800 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07"} err="failed to get container status \"862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07\": rpc error: code = NotFound desc = could not find container \"862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07\": container with ID starting with 862d6420c19251c1ac75af1340876907b0369f92bac0607baf523c5011668a07 not found: ID does not exist" Oct 31 12:53:40.162858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.162822 6727 scope.go:122] "RemoveContainer" containerID="29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230" Oct 31 12:53:40.172288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.172258 6727 scope.go:122] "RemoveContainer" containerID="29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230" Oct 31 12:53:40.172617 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:40.172593 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230\": container with ID starting with 29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230 not found: ID does not exist" containerID="29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230" Oct 31 12:53:40.172677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.172627 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230"} err="failed to get container status \"29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230\": rpc error: code = NotFound desc = could not find container \"29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230\": container with ID starting with 29724ed948ca7b35d8a2160298bcb04590b5df497c1e69260a11ceda4df78230 not found: ID does not exist" Oct 31 12:53:40.202473 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.202441 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc"] Oct 31 12:53:40.214915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.214892 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e98df8-6c87676b95-tbgdc"] Oct 31 12:53:40.231806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.231783 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs"] Oct 31 12:53:40.235309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.235291 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e98df8-686dd45497-t2fqs"] Oct 31 12:53:40.817834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:40.817801 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:41.304426 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:41.304393 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2d4864-c55c-4861-9b7e-24807e02d842" path="/var/lib/kubelet/pods/7b2d4864-c55c-4861-9b7e-24807e02d842/volumes" Oct 31 12:53:41.304917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:41.304894 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7409e4f-0b82-4884-ac1e-56a382f181bf" path="/var/lib/kubelet/pods/b7409e4f-0b82-4884-ac1e-56a382f181bf/volumes" Oct 31 12:53:41.649193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:41.649160 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:41.808817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:41.808784 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:42.432012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:42.431980 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:43.107482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:43.107444 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:43.264485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:43.257882 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:43.445120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:43.443159 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:43.540499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:43.539404 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l"] Oct 31 12:53:43.540499 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:43.540414 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" containerID="cri-o://1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232" gracePeriod=30 Oct 31 12:53:43.593894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:43.593857 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:43.884552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:43.878925 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:44.261285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:44.261192 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:44.332507 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:44.332473 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:53:44.658571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:44.658530 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:44.955304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:44.955146 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:45.188687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:45.188648 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:45.341719 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:45.341679 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:45.657591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:45.656804 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:46.654267 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:46.654232 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:47.920384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:47.920352 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:49.025369 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.025346 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:53:49.103615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.103580 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:49.159447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.159357 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-credential-socket\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.159756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.159734 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-data\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.160166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160145 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-socket\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.160390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160165 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-data" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:49.160519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160410 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-socket" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:49.160519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160426 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-token\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-token\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.160654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160609 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-envoy\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.160724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160661 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-certs\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.160724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160714 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-kube-api-access-m7bf9\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-kube-api-access-m7bf9\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.160826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160754 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istiod-ca-cert\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.160881 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160857 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-podinfo\") pod \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\" (UID: \"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b\") " Oct 31 12:53:49.161548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.160998 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-credential-socket" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:49.161548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.161296 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-certs" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:49.161548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.161411 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istiod-ca-cert" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:49.162274 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.162250 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.162274 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.162275 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.162431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.162290 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.162431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.162302 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.162431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.162316 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.165962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.165924 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd"] Oct 31 12:53:49.168929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.168905 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-envoy" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:53:49.202931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.201927 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-podinfo" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:53:49.203971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.203939 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-token" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:49.204108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.204034 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn"] Oct 31 12:53:49.207572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.207543 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-kube-api-access-m7bf9" pod "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" (UID: "5c34587a-5bfb-42a1-bad1-cc897fd2fc3b"). InnerVolumeSpecName "kube-api-access-m7bf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:49.243717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.243687 6727 generic.go:356] "Generic (PLEG): container finished" podID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerID="1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232" exitCode=0 Oct 31 12:53:49.243837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.243750 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" Oct 31 12:53:49.243837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.243770 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" event={"ID":"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b","Type":"ContainerDied","Data":"1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232"} Oct 31 12:53:49.243942 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.243835 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l" event={"ID":"5c34587a-5bfb-42a1-bad1-cc897fd2fc3b","Type":"ContainerDied","Data":"a054f59b21af8fd0dce903f49dce3340409fc18e07756c98afe0ee9e4ff053ef"} Oct 31 12:53:49.243942 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.243870 6727 scope.go:122] "RemoveContainer" containerID="1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232" Oct 31 12:53:49.257142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.257068 6727 scope.go:122] "RemoveContainer" containerID="9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69" Oct 31 12:53:49.264408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.264379 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.264594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.264415 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bf9\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-kube-api-access-m7bf9\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.264594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.264431 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.264594 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.264445 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:49.270001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.269982 6727 scope.go:122] "RemoveContainer" containerID="1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232" Oct 31 12:53:49.270317 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:49.270296 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232\": container with ID starting with 1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232 not found: ID does not exist" containerID="1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232" Oct 31 12:53:49.270396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.270333 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232"} err="failed to get container status \"1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232\": rpc error: code = NotFound desc = could not find container \"1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232\": container with ID starting with 1e3d7e565db038e599853ef2c4289d837d35d3d70272cd157742e220fec55232 not found: ID does not exist" Oct 31 12:53:49.270396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.270358 6727 scope.go:122] "RemoveContainer" containerID="9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69" Oct 31 12:53:49.270648 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:53:49.270625 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69\": container with ID starting with 9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69 not found: ID does not exist" containerID="9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69" Oct 31 12:53:49.270714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.270654 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69"} err="failed to get container status \"9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69\": rpc error: code = NotFound desc = could not find container \"9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69\": container with ID starting with 9ff900adf035fe9db307a8ec54e87948f23b1802130fbb2efa792132edd07d69 not found: ID does not exist" Oct 31 12:53:49.311805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.311776 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l"] Oct 31 12:53:49.316265 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.316245 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a0ea4-openshift-default-76d4959b8c-2vt7l"] Oct 31 12:53:49.401237 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.401199 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:49.613881 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.613851 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:49.914138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:49.914049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:50.257114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:50.257037 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:50.470403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:50.470370 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:50.678154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:50.678032 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:50.890326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:50.890292 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:51.179769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:51.179729 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:51.304036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:51.304000 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" path="/var/lib/kubelet/pods/5c34587a-5bfb-42a1-bad1-cc897fd2fc3b/volumes" Oct 31 12:53:51.479932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:51.479840 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:52.373201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:52.373168 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:53.275519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:53.268040 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:53.656676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:53.656643 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:53.809332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:53.809299 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:54.033124 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:54.033058 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:54.598609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:54.598579 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv"] Oct 31 12:53:54.598800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:54.598777 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" containerID="cri-o://6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390" gracePeriod=30 Oct 31 12:53:54.813154 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:54.806020 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:55.078612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:55.078581 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:55.289356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:55.289323 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:56.105358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:56.105317 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:56.350307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:56.350281 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:56.565234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:56.565116 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:56.605837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:56.605805 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q"] Oct 31 12:53:56.606136 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:56.606084 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" containerID="cri-o://f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f" gracePeriod=30 Oct 31 12:53:56.894494 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:56.886369 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:57.274150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:57.272883 6727 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:53:57.274150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:57.273027 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:53:57.638379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:57.638286 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:57.824131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:57.824090 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:58.519358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:58.519323 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:59.687191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:59.687153 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:59.962112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:53:59.962084 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:54:00.117161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.117087 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-istio-token\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-istio-token\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.117683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.117661 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-kube-api-access-ks8h2\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-kube-api-access-ks8h2\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.117830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.117779 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-certs\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.117948 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.117932 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-envoy\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.118255 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.118232 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0463c520-2269-4666-a30c-e6f36b31f750-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463c520-2269-4666-a30c-e6f36b31f750-istiod-ca-cert\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.118448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.118431 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-data\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.118775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.118536 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-credential-socket\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.118906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.118888 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0463c520-2269-4666-a30c-e6f36b31f750-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463c520-2269-4666-a30c-e6f36b31f750-istio-podinfo\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.118993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.118957 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-socket\") pod \"0463c520-2269-4666-a30c-e6f36b31f750\" (UID: \"0463c520-2269-4666-a30c-e6f36b31f750\") " Oct 31 12:54:00.119590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.119561 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-certs" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:00.119784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.119764 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-socket" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:00.119925 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.119902 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-data" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:00.120014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.119989 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-credential-socket" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:00.120122 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.120105 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.120185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.120132 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.120185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.120154 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0463c520-2269-4666-a30c-e6f36b31f750-istiod-ca-cert" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:00.121058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.121033 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-envoy" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:00.121141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.121089 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-istio-token" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:00.121280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.121258 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-kube-api-access-ks8h2" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "kube-api-access-ks8h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:00.121887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.121865 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0463c520-2269-4666-a30c-e6f36b31f750-istio-podinfo" pod "0463c520-2269-4666-a30c-e6f36b31f750" (UID: "0463c520-2269-4666-a30c-e6f36b31f750"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:00.221584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.221522 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.221584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.221545 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463c520-2269-4666-a30c-e6f36b31f750-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.221584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.221558 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.221584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.221570 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463c520-2269-4666-a30c-e6f36b31f750-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.221584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.221583 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463c520-2269-4666-a30c-e6f36b31f750-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.221802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.221595 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.221802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.221608 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8h2\" (UniqueName: \"kubernetes.io/projected/0463c520-2269-4666-a30c-e6f36b31f750-kube-api-access-ks8h2\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:00.343322 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:00.343272 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:54:00.369954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.369924 6727 generic.go:356] "Generic (PLEG): container finished" podID="0463c520-2269-4666-a30c-e6f36b31f750" containerID="6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390" exitCode=0 Oct 31 12:54:00.370130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.369975 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" Oct 31 12:54:00.370130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.370006 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" event={"ID":"0463c520-2269-4666-a30c-e6f36b31f750","Type":"ContainerDied","Data":"6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390"} Oct 31 12:54:00.370130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.370079 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv" event={"ID":"0463c520-2269-4666-a30c-e6f36b31f750","Type":"ContainerDied","Data":"9a2780db906199e966eceb26a85f755b4263a5b843d017a362ef5288878d8496"} Oct 31 12:54:00.370130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.370127 6727 scope.go:122] "RemoveContainer" containerID="6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390" Oct 31 12:54:00.382105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.382085 6727 scope.go:122] "RemoveContainer" containerID="78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53" Oct 31 12:54:00.392230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.392206 6727 scope.go:122] "RemoveContainer" containerID="6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390" Oct 31 12:54:00.392518 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:00.392496 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390\": container with ID starting with 6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390 not found: ID does not exist" containerID="6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390" Oct 31 12:54:00.392578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.392525 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390"} err="failed to get container status \"6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390\": rpc error: code = NotFound desc = could not find container \"6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390\": container with ID starting with 6d1146a3ca8c153895bb9912b5703666394f6ecc6ef1f554e37894aa29c9b390 not found: ID does not exist" Oct 31 12:54:00.392578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.392540 6727 scope.go:122] "RemoveContainer" containerID="78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53" Oct 31 12:54:00.392753 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:00.392735 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53\": container with ID starting with 78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53 not found: ID does not exist" containerID="78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53" Oct 31 12:54:00.392806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.392764 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53"} err="failed to get container status \"78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53\": rpc error: code = NotFound desc = could not find container \"78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53\": container with ID starting with 78c7203b52972d9865c1a1f5c562e300cfeb5d06f8a657d927a25b2057c36d53 not found: ID does not exist" Oct 31 12:54:00.400795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.400776 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv"] Oct 31 12:54:00.404986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.404967 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-fj8cv"] Oct 31 12:54:00.606259 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:00.606225 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:01.014491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:01.014397 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn"] Oct 31 12:54:01.304571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:01.304499 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0463c520-2269-4666-a30c-e6f36b31f750" path="/var/lib/kubelet/pods/0463c520-2269-4666-a30c-e6f36b31f750/volumes" Oct 31 12:54:01.396373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:01.396347 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:01.924429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:01.924393 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:54:02.055842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.055813 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57f48ec-1e88-4fb4-9569-7443b23107fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57f48ec-1e88-4fb4-9569-7443b23107fa-istiod-ca-cert\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.055906 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-credential-socket\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056067 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-podinfo\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056158 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-credential-socket" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:02.056279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056194 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57f48ec-1e88-4fb4-9569-7443b23107fa-istiod-ca-cert" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:02.056279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056256 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-kube-api-access-vwlx9\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-kube-api-access-vwlx9\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056348 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-data\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056396 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-token\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-token\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056417 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-socket\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056439 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-envoy\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056553 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-certs\") pod \"f57f48ec-1e88-4fb4-9569-7443b23107fa\" (UID: \"f57f48ec-1e88-4fb4-9569-7443b23107fa\") " Oct 31 12:54:02.056772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.056721 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-data" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:02.057057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.057029 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-certs" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:02.057167 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.057093 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-socket" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:02.057694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.057669 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.057800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.057699 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.057800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.057716 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.057800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.057731 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57f48ec-1e88-4fb4-9569-7443b23107fa-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.057800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.057746 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.058732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.058707 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-kube-api-access-vwlx9" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "kube-api-access-vwlx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:02.058848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.058809 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-podinfo" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:02.059173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.059102 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-envoy" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:02.059173 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.059125 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-token" pod "f57f48ec-1e88-4fb4-9569-7443b23107fa" (UID: "f57f48ec-1e88-4fb4-9569-7443b23107fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:02.159938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.159872 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.159938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.159902 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlx9\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-kube-api-access-vwlx9\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.159938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.159918 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.159938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.159931 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57f48ec-1e88-4fb4-9569-7443b23107fa-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:02.182882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.182846 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:02.427545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.424453 6727 generic.go:356] "Generic (PLEG): container finished" podID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerID="f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f" exitCode=0 Oct 31 12:54:02.427545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.424769 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" event={"ID":"f57f48ec-1e88-4fb4-9569-7443b23107fa","Type":"ContainerDied","Data":"f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f"} Oct 31 12:54:02.427545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.424822 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" event={"ID":"f57f48ec-1e88-4fb4-9569-7443b23107fa","Type":"ContainerDied","Data":"bcbe95fd18fc99072e629b8b659887bd519c0662f3694714acfb97235200c995"} Oct 31 12:54:02.427545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.424851 6727 scope.go:122] "RemoveContainer" containerID="f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f" Oct 31 12:54:02.427545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.425002 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q" Oct 31 12:54:02.475155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.475106 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:02.575975 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.572995 6727 scope.go:122] "RemoveContainer" containerID="916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e" Oct 31 12:54:02.614485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.608659 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q"] Oct 31 12:54:02.614485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.613458 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1a7d0e-openshift-default-84bd956545-tb64q"] Oct 31 12:54:02.627048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.627026 6727 scope.go:122] "RemoveContainer" containerID="f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f" Oct 31 12:54:02.627420 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:02.627396 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f\": container with ID starting with f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f not found: ID does not exist" containerID="f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f" Oct 31 12:54:02.627525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.627428 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f"} err="failed to get container status \"f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f\": rpc error: code = NotFound desc = could not find container \"f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f\": container with ID starting with f67dee2cdaa5d30ea6c82b60c637b08a4ba01acc35f8853ef29e5f1120233b2f not found: ID does not exist" Oct 31 12:54:02.627525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.627449 6727 scope.go:122] "RemoveContainer" containerID="916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e" Oct 31 12:54:02.627781 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:02.627760 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e\": container with ID starting with 916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e not found: ID does not exist" containerID="916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e" Oct 31 12:54:02.627840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:02.627789 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e"} err="failed to get container status \"916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e\": rpc error: code = NotFound desc = could not find container \"916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e\": container with ID starting with 916e775a9021998c7159b458bc968bee079777d4c9acf57ebb308bdd0764895e not found: ID does not exist" Oct 31 12:54:02.674575 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:02.674543 6727 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57f48ec_1e88_4fb4_9569_7443b23107fa.slice/crio-bcbe95fd18fc99072e629b8b659887bd519c0662f3694714acfb97235200c995\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57f48ec_1e88_4fb4_9569_7443b23107fa.slice\": RecentStats: unable to find data in memory cache]" Oct 31 12:54:03.303812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.303780 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" path="/var/lib/kubelet/pods/f57f48ec-1e88-4fb4-9569-7443b23107fa/volumes" Oct 31 12:54:03.670690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.670658 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:03.674231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.674202 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6"] Oct 31 12:54:03.675253 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675233 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675254 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675282 6727 cpu_manager.go:414] "containerMap: removing container" podUID="b7409e4f-0b82-4884-ac1e-56a382f181bf" containerName="payload-pre-processing" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675291 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7409e4f-0b82-4884-ac1e-56a382f181bf" containerName="payload-pre-processing" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675322 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675330 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675356 6727 cpu_manager.go:414] "containerMap: removing container" podUID="3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" containerName="payload-pre-processing" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675363 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" containerName="payload-pre-processing" Oct 31 12:54:03.675381 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675380 6727 cpu_manager.go:414] "containerMap: removing container" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675386 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675406 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675412 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675422 6727 cpu_manager.go:414] "containerMap: removing container" podUID="7b2d4864-c55c-4861-9b7e-24807e02d842" containerName="payload-processing" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675429 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2d4864-c55c-4861-9b7e-24807e02d842" containerName="payload-processing" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675459 6727 cpu_manager.go:414] "containerMap: removing container" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675481 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675516 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675523 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675723 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd39810-98b2-4aac-9fa9-1d2d56f25c2f" containerName="payload-pre-processing" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675736 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="560131f7-efd5-4343-8378-6efcb60cc69c" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675747 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675756 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2d4864-c55c-4861-9b7e-24807e02d842" containerName="payload-processing" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675771 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675783 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675797 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7409e4f-0b82-4884-ac1e-56a382f181bf" containerName="payload-pre-processing" Oct 31 12:54:03.675826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.675809 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c34587a-5bfb-42a1-bad1-cc897fd2fc3b" containerName="istio-proxy" Oct 31 12:54:03.681112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.681087 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:03.704932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.704909 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6"] Oct 31 12:54:03.717895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.717874 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-71bc05-dockercfg-m74mt\"" Oct 31 12:54:03.782114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.782083 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-71bc05\"" Oct 31 12:54:03.813491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.812678 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfgf\" (UniqueName: \"kubernetes.io/projected/2e47b7b7-859b-4806-8829-b41130e55cb3-kube-api-access-wvfgf\") pod \"payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:03.813491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.812757 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e47b7b7-859b-4806-8829-b41130e55cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:03.919953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.914810 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfgf\" (UniqueName: \"kubernetes.io/projected/2e47b7b7-859b-4806-8829-b41130e55cb3-kube-api-access-wvfgf\") pod \"payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:03.919953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.914879 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e47b7b7-859b-4806-8829-b41130e55cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:03.919953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.916197 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e47b7b7-859b-4806-8829-b41130e55cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:03.960535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.955891 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:03.991491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:03.990294 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfgf\" (UniqueName: \"kubernetes.io/projected/2e47b7b7-859b-4806-8829-b41130e55cb3-kube-api-access-wvfgf\") pod \"payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:04.015226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.013895 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:04.220083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.220010 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" containerID="cri-o://3fecc22682efbaa15b9238b27e749e8bb7944f82f8c55f7e5df775adb3ab403f" gracePeriod=45 Oct 31 12:54:04.226645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.226607 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" containerID="cri-o://97d8c53aab1f2cd7db2c3e271bd5a14acf700daea9443dda8f4011c5dd61b504" gracePeriod=45 Oct 31 12:54:04.343601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.343566 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": dial tcp 10.134.0.58:8000: connect: connection refused" Oct 31 12:54:04.405432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.405402 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6"] Oct 31 12:54:04.406866 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:54:04.406047 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e47b7b7_859b_4806_8829_b41130e55cb3.slice/crio-9c4069180bc11825ff8f4f071ff4534b060635a3d4ccb145e5e5673ae96cf289 WatchSource:0}: Error finding container 9c4069180bc11825ff8f4f071ff4534b060635a3d4ccb145e5e5673ae96cf289: Status 404 returned error can't find the container with id 9c4069180bc11825ff8f4f071ff4534b060635a3d4ccb145e5e5673ae96cf289 Oct 31 12:54:04.464333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.464302 6727 generic.go:356] "Generic (PLEG): container finished" podID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerID="97d8c53aab1f2cd7db2c3e271bd5a14acf700daea9443dda8f4011c5dd61b504" exitCode=0 Oct 31 12:54:04.464458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.464437 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" event={"ID":"69da0e2c-7478-499f-879e-20aa90d6ad89","Type":"ContainerDied","Data":"97d8c53aab1f2cd7db2c3e271bd5a14acf700daea9443dda8f4011c5dd61b504"} Oct 31 12:54:04.466345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.466316 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" event={"ID":"2e47b7b7-859b-4806-8829-b41130e55cb3","Type":"ContainerStarted","Data":"9c4069180bc11825ff8f4f071ff4534b060635a3d4ccb145e5e5673ae96cf289"} Oct 31 12:54:04.469534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.469498 6727 generic.go:356] "Generic (PLEG): container finished" podID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerID="3fecc22682efbaa15b9238b27e749e8bb7944f82f8c55f7e5df775adb3ab403f" exitCode=0 Oct 31 12:54:04.469648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.469635 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" event={"ID":"54d09a8c-25dd-4644-8455-0e9c5ebed221","Type":"ContainerDied","Data":"3fecc22682efbaa15b9238b27e749e8bb7944f82f8c55f7e5df775adb3ab403f"} Oct 31 12:54:04.535747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.535723 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:54:04.555694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.555665 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54d09a8c-25dd-4644-8455-0e9c5ebed221-tls-certs\" (UniqueName: \"kubernetes.io/secret/54d09a8c-25dd-4644-8455-0e9c5ebed221-tls-certs\") pod \"54d09a8c-25dd-4644-8455-0e9c5ebed221\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " Oct 31 12:54:04.555822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.555713 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-model-cache\") pod \"54d09a8c-25dd-4644-8455-0e9c5ebed221\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " Oct 31 12:54:04.555889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.555820 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-home\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-home\") pod \"54d09a8c-25dd-4644-8455-0e9c5ebed221\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " Oct 31 12:54:04.555949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.555907 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-dshm\") pod \"54d09a8c-25dd-4644-8455-0e9c5ebed221\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " Oct 31 12:54:04.555949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.555938 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d09a8c-25dd-4644-8455-0e9c5ebed221-kube-api-access-wtmqb\" (UniqueName: \"kubernetes.io/projected/54d09a8c-25dd-4644-8455-0e9c5ebed221-kube-api-access-wtmqb\") pod \"54d09a8c-25dd-4644-8455-0e9c5ebed221\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " Oct 31 12:54:04.556096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.556081 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-tmp-dir\") pod \"54d09a8c-25dd-4644-8455-0e9c5ebed221\" (UID: \"54d09a8c-25dd-4644-8455-0e9c5ebed221\") " Oct 31 12:54:04.556229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.556204 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-home" pod "54d09a8c-25dd-4644-8455-0e9c5ebed221" (UID: "54d09a8c-25dd-4644-8455-0e9c5ebed221"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.556477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.556442 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-model-cache" pod "54d09a8c-25dd-4644-8455-0e9c5ebed221" (UID: "54d09a8c-25dd-4644-8455-0e9c5ebed221"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.556765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.556741 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-tmp-dir" pod "54d09a8c-25dd-4644-8455-0e9c5ebed221" (UID: "54d09a8c-25dd-4644-8455-0e9c5ebed221"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.557701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.557627 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.557701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.557656 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.557701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.557672 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.558312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.558239 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d09a8c-25dd-4644-8455-0e9c5ebed221-tls-certs" pod "54d09a8c-25dd-4644-8455-0e9c5ebed221" (UID: "54d09a8c-25dd-4644-8455-0e9c5ebed221"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:04.560763 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.559859 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-dshm" pod "54d09a8c-25dd-4644-8455-0e9c5ebed221" (UID: "54d09a8c-25dd-4644-8455-0e9c5ebed221"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.560868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.560754 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d09a8c-25dd-4644-8455-0e9c5ebed221-kube-api-access-wtmqb" pod "54d09a8c-25dd-4644-8455-0e9c5ebed221" (UID: "54d09a8c-25dd-4644-8455-0e9c5ebed221"). InnerVolumeSpecName "kube-api-access-wtmqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:04.596261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.596222 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:54:04.658798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.658762 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-home\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-home\") pod \"69da0e2c-7478-499f-879e-20aa90d6ad89\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " Oct 31 12:54:04.658969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.658871 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69da0e2c-7478-499f-879e-20aa90d6ad89-tls-certs\" (UniqueName: \"kubernetes.io/secret/69da0e2c-7478-499f-879e-20aa90d6ad89-tls-certs\") pod \"69da0e2c-7478-499f-879e-20aa90d6ad89\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " Oct 31 12:54:04.659041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.659024 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69da0e2c-7478-499f-879e-20aa90d6ad89-kube-api-access-5vgkf\" (UniqueName: \"kubernetes.io/projected/69da0e2c-7478-499f-879e-20aa90d6ad89-kube-api-access-5vgkf\") pod \"69da0e2c-7478-499f-879e-20aa90d6ad89\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " Oct 31 12:54:04.659102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.659041 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-home" pod "69da0e2c-7478-499f-879e-20aa90d6ad89" (UID: "69da0e2c-7478-499f-879e-20aa90d6ad89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.659102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.659063 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-dshm\") pod \"69da0e2c-7478-499f-879e-20aa90d6ad89\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " Oct 31 12:54:04.659213 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.659162 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-tmp-dir\") pod \"69da0e2c-7478-499f-879e-20aa90d6ad89\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " Oct 31 12:54:04.659334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.659313 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-model-cache\") pod \"69da0e2c-7478-499f-879e-20aa90d6ad89\" (UID: \"69da0e2c-7478-499f-879e-20aa90d6ad89\") " Oct 31 12:54:04.659601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.659573 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-model-cache" pod "69da0e2c-7478-499f-879e-20aa90d6ad89" (UID: "69da0e2c-7478-499f-879e-20aa90d6ad89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.659601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.659586 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-tmp-dir" pod "69da0e2c-7478-499f-879e-20aa90d6ad89" (UID: "69da0e2c-7478-499f-879e-20aa90d6ad89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.660230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.660202 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d09a8c-25dd-4644-8455-0e9c5ebed221-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.660358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.660233 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.660358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.660247 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d09a8c-25dd-4644-8455-0e9c5ebed221-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.660358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.660260 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtmqb\" (UniqueName: \"kubernetes.io/projected/54d09a8c-25dd-4644-8455-0e9c5ebed221-kube-api-access-wtmqb\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.660358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.660277 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.660358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.660291 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.661538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.661508 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69da0e2c-7478-499f-879e-20aa90d6ad89-tls-certs" pod "69da0e2c-7478-499f-879e-20aa90d6ad89" (UID: "69da0e2c-7478-499f-879e-20aa90d6ad89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:04.661894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.661869 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-dshm" pod "69da0e2c-7478-499f-879e-20aa90d6ad89" (UID: "69da0e2c-7478-499f-879e-20aa90d6ad89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:04.663253 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.663231 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69da0e2c-7478-499f-879e-20aa90d6ad89-kube-api-access-5vgkf" pod "69da0e2c-7478-499f-879e-20aa90d6ad89" (UID: "69da0e2c-7478-499f-879e-20aa90d6ad89"). InnerVolumeSpecName "kube-api-access-5vgkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:04.762610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.762539 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69da0e2c-7478-499f-879e-20aa90d6ad89-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.762610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.762571 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vgkf\" (UniqueName: \"kubernetes.io/projected/69da0e2c-7478-499f-879e-20aa90d6ad89-kube-api-access-5vgkf\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.762610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.762582 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69da0e2c-7478-499f-879e-20aa90d6ad89-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:04.972801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:04.972756 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:05.479817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.479790 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" Oct 31 12:54:05.480268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.479816 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd" event={"ID":"54d09a8c-25dd-4644-8455-0e9c5ebed221","Type":"ContainerDied","Data":"cdb605105b567431c69808f9dfdce9fffce2421cd8481c038d74241f3c08fb9d"} Oct 31 12:54:05.480268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.479868 6727 scope.go:122] "RemoveContainer" containerID="3fecc22682efbaa15b9238b27e749e8bb7944f82f8c55f7e5df775adb3ab403f" Oct 31 12:54:05.481864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.481834 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" event={"ID":"69da0e2c-7478-499f-879e-20aa90d6ad89","Type":"ContainerDied","Data":"76a13a736411306695284a2279b44624428c1c56d4ffdd059cfd0b501ae53b78"} Oct 31 12:54:05.482053 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.481917 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn" Oct 31 12:54:05.485131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.484590 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" event={"ID":"2e47b7b7-859b-4806-8829-b41130e55cb3","Type":"ContainerStarted","Data":"f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60"} Oct 31 12:54:05.485131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.484741 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:05.492193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.492176 6727 scope.go:122] "RemoveContainer" containerID="97d8c53aab1f2cd7db2c3e271bd5a14acf700daea9443dda8f4011c5dd61b504" Oct 31 12:54:05.521656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.521633 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn"] Oct 31 12:54:05.543950 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.543922 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-w1-85ce08-kserve-6d7cf8c559-52cqn"] Oct 31 12:54:05.612302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.612256 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" podStartSLOduration=2.612243689 podStartE2EDuration="2.612243689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:05.585821454 +0000 UTC m=+1556.400835845" watchObservedRunningTime="2026-10-31 12:54:05.612243689 +0000 UTC m=+1556.427258079" Oct 31 12:54:05.612505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.612490 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd"] Oct 31 12:54:05.625192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.625166 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/distinct-2-w1-85ce08-kserve-c67b669dd-g9gcd"] Oct 31 12:54:05.927368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:05.927247 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:06.640356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:06.640323 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:07.267862 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:07.267831 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr"] Oct 31 12:54:07.325717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:07.325671 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" path="/var/lib/kubelet/pods/54d09a8c-25dd-4644-8455-0e9c5ebed221/volumes" Oct 31 12:54:07.326710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:07.326686 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" path="/var/lib/kubelet/pods/69da0e2c-7478-499f-879e-20aa90d6ad89/volumes" Oct 31 12:54:07.369126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:07.369076 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:07.537715 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:07.537599 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:07.539418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:07.539383 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk"] Oct 31 12:54:07.687599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:07.687568 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:08.529662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:08.529621 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:08.720387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:08.720361 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:09.272177 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:09.272142 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:09.740496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:09.740431 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb"] Oct 31 12:54:09.740908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:09.740632 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" containerID="cri-o://6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09" gracePeriod=30 Oct 31 12:54:10.086581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:10.086533 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:10.772258 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:10.754677 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:11.437512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:11.437440 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:12.146628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.146595 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:12.268509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.268471 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" podUID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" containerName="payload-pre-processing" containerID="cri-o://98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9" gracePeriod=25 Oct 31 12:54:12.540097 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.540024 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" podUID="46f3751c-f307-4903-ab02-f236dc860401" containerName="payload-pre-processing" containerID="cri-o://a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a" gracePeriod=25 Oct 31 12:54:12.828213 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.828183 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:54:12.869715 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.869686 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f3751c-f307-4903-ab02-f236dc860401-kube-api-access-fxdqj\" (UniqueName: \"kubernetes.io/projected/46f3751c-f307-4903-ab02-f236dc860401-kube-api-access-fxdqj\") pod \"46f3751c-f307-4903-ab02-f236dc860401\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " Oct 31 12:54:12.869865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.869852 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f3751c-f307-4903-ab02-f236dc860401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f3751c-f307-4903-ab02-f236dc860401-plugins-config-volume\") pod \"46f3751c-f307-4903-ab02-f236dc860401\" (UID: \"46f3751c-f307-4903-ab02-f236dc860401\") " Oct 31 12:54:12.870271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.870240 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f3751c-f307-4903-ab02-f236dc860401-plugins-config-volume" pod "46f3751c-f307-4903-ab02-f236dc860401" (UID: "46f3751c-f307-4903-ab02-f236dc860401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:12.870690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.870673 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f3751c-f307-4903-ab02-f236dc860401-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:12.871778 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.871752 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f3751c-f307-4903-ab02-f236dc860401-kube-api-access-fxdqj" pod "46f3751c-f307-4903-ab02-f236dc860401" (UID: "46f3751c-f307-4903-ab02-f236dc860401"). InnerVolumeSpecName "kube-api-access-fxdqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:12.911746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.911704 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:12.974845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:12.974809 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxdqj\" (UniqueName: \"kubernetes.io/projected/46f3751c-f307-4903-ab02-f236dc860401-kube-api-access-fxdqj\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:13.560692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.560614 6727 generic.go:356] "Generic (PLEG): container finished" podID="46f3751c-f307-4903-ab02-f236dc860401" containerID="a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a" exitCode=0 Oct 31 12:54:13.560692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.560656 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" Oct 31 12:54:13.561162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.560690 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" event={"ID":"46f3751c-f307-4903-ab02-f236dc860401","Type":"ContainerDied","Data":"a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a"} Oct 31 12:54:13.561162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.560738 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk" event={"ID":"46f3751c-f307-4903-ab02-f236dc860401","Type":"ContainerDied","Data":"32a3e7b80c2fd326eba97ed7e8769c375cd4229216214450d2d286267563729f"} Oct 31 12:54:13.561162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.560763 6727 scope.go:122] "RemoveContainer" containerID="a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a" Oct 31 12:54:13.571740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.571721 6727 scope.go:122] "RemoveContainer" containerID="a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a" Oct 31 12:54:13.571975 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:13.571957 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a\": container with ID starting with a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a not found: ID does not exist" containerID="a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a" Oct 31 12:54:13.572032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.571988 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a"} err="failed to get container status \"a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a\": rpc error: code = NotFound desc = could not find container \"a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a\": container with ID starting with a80fc68ee6d026c694977774a924fd0a3baff5f72c846eb86a751fb05949f43a not found: ID does not exist" Oct 31 12:54:13.595395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.595362 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk"] Oct 31 12:54:13.609683 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.609662 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb2b1d-7dd977f9-qspqk"] Oct 31 12:54:13.715996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.715969 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:13.917768 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:13.917735 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:14.063107 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:14.063074 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:14.327046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:14.327021 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:54:14.327207 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:14.327074 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:54:14.863673 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:14.863643 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" podUID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:54:14.894560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:14.887891 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:15.063096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.063010 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:15.291279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.291244 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:54:15.305627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.305596 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f3751c-f307-4903-ab02-f236dc860401" path="/var/lib/kubelet/pods/46f3751c-f307-4903-ab02-f236dc860401/volumes" Oct 31 12:54:15.356379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.356342 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-certs\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.356606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.356441 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-envoy\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.356606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.356533 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-socket\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.356606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.356580 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-data\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.356804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.356747 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-credential-socket\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.357838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.356855 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-kube-api-access-q78np\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-kube-api-access-q78np\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.357838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.357005 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-istio-token\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-istio-token\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.357838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.357042 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771d25b0-f843-4e87-8424-cfbb691f3173-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771d25b0-f843-4e87-8424-cfbb691f3173-istiod-ca-cert\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.357838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.357150 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/771d25b0-f843-4e87-8424-cfbb691f3173-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771d25b0-f843-4e87-8424-cfbb691f3173-istio-podinfo\") pod \"771d25b0-f843-4e87-8424-cfbb691f3173\" (UID: \"771d25b0-f843-4e87-8424-cfbb691f3173\") " Oct 31 12:54:15.357838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.357327 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-data" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:15.357838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.357523 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-certs" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:15.358312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.358288 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.359553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.359532 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.361788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.361765 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-socket" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:15.362449 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.362419 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/771d25b0-f843-4e87-8424-cfbb691f3173-istio-podinfo" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:15.362801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.362778 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771d25b0-f843-4e87-8424-cfbb691f3173-istiod-ca-cert" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:15.363139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.363115 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-credential-socket" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:15.369212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.369188 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-istio-token" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:15.371982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.371918 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-envoy" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:15.374894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.374872 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-kube-api-access-q78np" pod "771d25b0-f843-4e87-8424-cfbb691f3173" (UID: "771d25b0-f843-4e87-8424-cfbb691f3173"). InnerVolumeSpecName "kube-api-access-q78np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:15.461709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.461681 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.461709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.461709 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.461905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.461719 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q78np\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-kube-api-access-q78np\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.461905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.461730 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771d25b0-f843-4e87-8424-cfbb691f3173-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.461905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.461739 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771d25b0-f843-4e87-8424-cfbb691f3173-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.461905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.461749 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771d25b0-f843-4e87-8424-cfbb691f3173-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.461905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.461757 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771d25b0-f843-4e87-8424-cfbb691f3173-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:15.610084 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.609927 6727 generic.go:356] "Generic (PLEG): container finished" podID="771d25b0-f843-4e87-8424-cfbb691f3173" containerID="6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09" exitCode=0 Oct 31 12:54:15.610291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.610238 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" event={"ID":"771d25b0-f843-4e87-8424-cfbb691f3173","Type":"ContainerDied","Data":"6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09"} Oct 31 12:54:15.610361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.610289 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" event={"ID":"771d25b0-f843-4e87-8424-cfbb691f3173","Type":"ContainerDied","Data":"647fbc9844ecbdf39bfa950c3b86ecdb15e2bec48ef596f93226cca5ba69bdae"} Oct 31 12:54:15.610361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.610315 6727 scope.go:122] "RemoveContainer" containerID="6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09" Oct 31 12:54:15.615483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.610661 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb" Oct 31 12:54:15.640503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.638748 6727 scope.go:122] "RemoveContainer" containerID="6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09" Oct 31 12:54:15.640503 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:15.639127 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09\": container with ID starting with 6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09 not found: ID does not exist" containerID="6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09" Oct 31 12:54:15.640503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.639159 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09"} err="failed to get container status \"6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09\": rpc error: code = NotFound desc = could not find container \"6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09\": container with ID starting with 6d627340946cad537b504738c34e24e806ccdf01c3467f7e42fbd4d248e27f09 not found: ID does not exist" Oct 31 12:54:15.689315 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.689281 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:15.693100 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.693070 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb"] Oct 31 12:54:15.699066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:15.699032 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-tdzwb"] Oct 31 12:54:16.004944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.004735 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:16.036899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.036862 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" containerID="cri-o://09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71" gracePeriod=45 Oct 31 12:54:16.404854 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.398722 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:54:16.501514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.500899 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:54:16.511518 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.511488 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-tmp-dir\") pod \"06585f3d-576a-4abd-94b5-1ae90777820b\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " Oct 31 12:54:16.511696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.511526 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-dshm\") pod \"06585f3d-576a-4abd-94b5-1ae90777820b\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " Oct 31 12:54:16.511696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.511572 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-home\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-home\") pod \"06585f3d-576a-4abd-94b5-1ae90777820b\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " Oct 31 12:54:16.511823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.511716 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06585f3d-576a-4abd-94b5-1ae90777820b-tls-certs\" (UniqueName: \"kubernetes.io/secret/06585f3d-576a-4abd-94b5-1ae90777820b-tls-certs\") pod \"06585f3d-576a-4abd-94b5-1ae90777820b\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " Oct 31 12:54:16.511927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.511911 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-model-cache\") pod \"06585f3d-576a-4abd-94b5-1ae90777820b\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " Oct 31 12:54:16.512055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.512042 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06585f3d-576a-4abd-94b5-1ae90777820b-kube-api-access-d6h49\" (UniqueName: \"kubernetes.io/projected/06585f3d-576a-4abd-94b5-1ae90777820b-kube-api-access-d6h49\") pod \"06585f3d-576a-4abd-94b5-1ae90777820b\" (UID: \"06585f3d-576a-4abd-94b5-1ae90777820b\") " Oct 31 12:54:16.512962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.512931 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-model-cache" pod "06585f3d-576a-4abd-94b5-1ae90777820b" (UID: "06585f3d-576a-4abd-94b5-1ae90777820b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:16.513178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.513157 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-home" pod "06585f3d-576a-4abd-94b5-1ae90777820b" (UID: "06585f3d-576a-4abd-94b5-1ae90777820b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:16.513511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.513448 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-tmp-dir" pod "06585f3d-576a-4abd-94b5-1ae90777820b" (UID: "06585f3d-576a-4abd-94b5-1ae90777820b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:16.515531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.515505 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06585f3d-576a-4abd-94b5-1ae90777820b-tls-certs" pod "06585f3d-576a-4abd-94b5-1ae90777820b" (UID: "06585f3d-576a-4abd-94b5-1ae90777820b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:16.515718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.515705 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-dshm" pod "06585f3d-576a-4abd-94b5-1ae90777820b" (UID: "06585f3d-576a-4abd-94b5-1ae90777820b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:16.519921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.519894 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06585f3d-576a-4abd-94b5-1ae90777820b-kube-api-access-d6h49" pod "06585f3d-576a-4abd-94b5-1ae90777820b" (UID: "06585f3d-576a-4abd-94b5-1ae90777820b"). InnerVolumeSpecName "kube-api-access-d6h49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:16.538546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.535497 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:16.622720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.622681 6727 generic.go:356] "Generic (PLEG): container finished" podID="06585f3d-576a-4abd-94b5-1ae90777820b" containerID="09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71" exitCode=0 Oct 31 12:54:16.622720 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.622725 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" Oct 31 12:54:16.623203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.622853 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" event={"ID":"06585f3d-576a-4abd-94b5-1ae90777820b","Type":"ContainerDied","Data":"09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71"} Oct 31 12:54:16.623203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.622890 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" event={"ID":"06585f3d-576a-4abd-94b5-1ae90777820b","Type":"ContainerDied","Data":"6a49cae092164dec4cfed51f7f3e3b147dd4a7a385d4816e9c2f3e0dc6c3ff51"} Oct 31 12:54:16.623203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.622914 6727 scope.go:122] "RemoveContainer" containerID="09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71" Oct 31 12:54:16.627949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.625756 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06585f3d-576a-4abd-94b5-1ae90777820b-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:16.627949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.625785 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:16.627949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.625803 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6h49\" (UniqueName: \"kubernetes.io/projected/06585f3d-576a-4abd-94b5-1ae90777820b-kube-api-access-d6h49\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:16.627949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.625818 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:16.627949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.625833 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:16.627949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.625846 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06585f3d-576a-4abd-94b5-1ae90777820b-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:16.679551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.677333 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn"] Oct 31 12:54:16.684493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.684451 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn"] Oct 31 12:54:16.702379 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.702084 6727 scope.go:122] "RemoveContainer" containerID="09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71" Oct 31 12:54:16.702650 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:16.702594 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71\": container with ID starting with 09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71 not found: ID does not exist" containerID="09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71" Oct 31 12:54:16.702749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.702663 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71"} err="failed to get container status \"09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71\": rpc error: code = NotFound desc = could not find container \"09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71\": container with ID starting with 09c1f1f5ea894f0cc4d582582fc4afc3979288f501d07fecc5b660d206269a71 not found: ID does not exist" Oct 31 12:54:16.853860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:16.853829 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:17.249482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:17.243127 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-2-w3-b218d3-kserve-797ff7c57c-gdfnn" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:54:17.358576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:17.358537 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" path="/var/lib/kubelet/pods/06585f3d-576a-4abd-94b5-1ae90777820b/volumes" Oct 31 12:54:17.382497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:17.363450 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" path="/var/lib/kubelet/pods/771d25b0-f843-4e87-8424-cfbb691f3173/volumes" Oct 31 12:54:17.995398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:17.995358 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:18.383486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:18.379005 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:18.551538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:18.547413 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:18.560611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:18.559838 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw"] Oct 31 12:54:18.560611 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:18.560049 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" podUID="9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" containerName="maas-api" containerID="cri-o://9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1" gracePeriod=30 Oct 31 12:54:18.726535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:18.725916 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:19.003027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.002939 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:19.441552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.435992 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:19.557346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.557320 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:54:19.658783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.658745 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:19.755300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.747776 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-kube-api-access-4zwn9\" (UniqueName: \"kubernetes.io/projected/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-kube-api-access-4zwn9\") pod \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " Oct 31 12:54:19.755300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.747889 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-maas-api-tls\") pod \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " Oct 31 12:54:19.755300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.748074 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-metrics-certs\") pod \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " Oct 31 12:54:19.755300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.748237 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-openshift-service-ca\") pod \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\" (UID: \"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3\") " Oct 31 12:54:19.756865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.756817 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-openshift-service-ca" pod "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" (UID: "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:19.808018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.787495 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:19.838006 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.837973 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-kube-api-access-4zwn9" pod "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" (UID: "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3"). InnerVolumeSpecName "kube-api-access-4zwn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:19.845530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.845501 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:19.847125 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.846936 6727 generic.go:356] "Generic (PLEG): container finished" podID="9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" containerID="9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1" exitCode=0 Oct 31 12:54:19.847997 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.846979 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" event={"ID":"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3","Type":"ContainerDied","Data":"9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1"} Oct 31 12:54:19.848168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.848149 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" event={"ID":"9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3","Type":"ContainerDied","Data":"c32aefcad6c20db356ddb73b00b424b36c1a01da03284b63fdd0d703385a3502"} Oct 31 12:54:19.848297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.848284 6727 scope.go:122] "RemoveContainer" containerID="9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1" Oct 31 12:54:19.848551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.847338 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw" Oct 31 12:54:19.903543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.899335 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zwn9\" (UniqueName: \"kubernetes.io/projected/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-kube-api-access-4zwn9\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:19.928010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.926672 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-maas-api-tls" pod "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" (UID: "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:19.928010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.926756 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-metrics-certs" pod "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" (UID: "9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:19.942253 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.942233 6727 scope.go:122] "RemoveContainer" containerID="9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1" Oct 31 12:54:19.942627 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:19.942607 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1\": container with ID starting with 9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1 not found: ID does not exist" containerID="9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1" Oct 31 12:54:19.942722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:19.942636 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1"} err="failed to get container status \"9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1\": rpc error: code = NotFound desc = could not find container \"9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1\": container with ID starting with 9ec924090afca194264b33484ceadfd5ecb57a202a1b36cdd1fd9fd1d75213a1 not found: ID does not exist" Oct 31 12:54:20.003485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:20.003180 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:20.003485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:20.003218 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:20.221224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:20.221156 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw"] Oct 31 12:54:20.228520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:20.227621 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85ce08-6458dc558c-gb7qw"] Oct 31 12:54:20.711485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:20.688049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:20.873335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:20.873298 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:21.116289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:21.116247 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:21.455227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:21.454897 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" path="/var/lib/kubelet/pods/9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3/volumes" Oct 31 12:54:21.709538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:21.708147 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:21.877488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:21.871975 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:22.823862 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:22.823823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:23.922088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:23.921989 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:24.582234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:24.579145 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:24.863282 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:24.863213 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" podUID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:54:24.984706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:24.984675 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv"] Oct 31 12:54:24.985164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:24.984921 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" containerID="cri-o://3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28" gracePeriod=30 Oct 31 12:54:25.725170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:25.725124 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:26.208186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:26.208154 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:26.624706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:26.624633 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:26.852775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:26.852716 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:27.017786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:27.017695 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:28.036775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:28.036747 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:28.704037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:28.704004 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:28.868631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:28.868606 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:29.043944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.043917 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7"] Oct 31 12:54:29.044303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.044115 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" containerID="cri-o://32931eac714d01b04d5d5e9b3b8c04a14bb27717ec3e216e42b6ab91379fe479" gracePeriod=30 Oct 31 12:54:29.044944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.044920 6727 generic.go:356] "Generic (PLEG): container finished" podID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerID="475c06dc335837dbd5890eed6c40fd6c7a69a2c25d26216fe9103d845df196f3" exitCode=0 Oct 31 12:54:29.045086 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.044992 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" event={"ID":"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d","Type":"ContainerDied","Data":"475c06dc335837dbd5890eed6c40fd6c7a69a2c25d26216fe9103d845df196f3"} Oct 31 12:54:29.045633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.045612 6727 scope.go:122] "RemoveContainer" containerID="475c06dc335837dbd5890eed6c40fd6c7a69a2c25d26216fe9103d845df196f3" Oct 31 12:54:29.047645 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.047627 6727 generic.go:356] "Generic (PLEG): container finished" podID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerID="baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a" exitCode=0 Oct 31 12:54:29.047731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.047680 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" event={"ID":"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa","Type":"ContainerDied","Data":"baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a"} Oct 31 12:54:29.048164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.048145 6727 scope.go:122] "RemoveContainer" containerID="baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a" Oct 31 12:54:29.049077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.049045 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:54:29.049171 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.049130 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:54:29.049240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.049177 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:54:29.050189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.050014 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:54:29.050189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.050135 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:54:29.050385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.050202 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:54:29.069022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.068999 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:29.079837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.079021 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw"] Oct 31 12:54:29.079837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.079244 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" containerID="cri-o://630d80cb32b11580807edcca86f400263dddbda3907daa79b926e3e3a61b3d5e" gracePeriod=30 Oct 31 12:54:29.094048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.094015 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2"] Oct 31 12:54:29.094332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.094291 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" containerID="cri-o://941f703504ebcb532e4e40b0711e5fb3020834439274682bd4b065775c9dc031" gracePeriod=30 Oct 31 12:54:29.301572 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.291114 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:29.584495 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.584441 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:54:29.584695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:29.584519 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:54:30.057450 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.057420 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" event={"ID":"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d","Type":"ContainerStarted","Data":"cd81376acc6d238b83d584633c09a56a6e7429815f9dec7b6e4e164eab3da88f"} Oct 31 12:54:30.060751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.060724 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" event={"ID":"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa","Type":"ContainerStarted","Data":"dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103"} Oct 31 12:54:30.352424 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:30.352382 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:54:30.410876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.410785 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.525728 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-podinfo\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.525879 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-token\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-token\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.525921 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istiod-ca-cert\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.526082 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-envoy\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.526117 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-kube-api-access-95cvp\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-kube-api-access-95cvp\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.526149 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-data\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.526269 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-credential-socket\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.526353 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-socket\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.526527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.526395 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-certs\") pod \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\" (UID: \"2b7edc1d-99bc-4df6-93b2-bc945072d2ed\") " Oct 31 12:54:30.529798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.529222 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istiod-ca-cert" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:30.529798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.529539 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-data" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:30.530352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.530019 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-certs" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:30.530728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.530528 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-credential-socket" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:30.530900 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.530855 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-socket" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:30.533847 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.533808 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-token" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:30.536145 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.536112 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-podinfo" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:30.536534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.536458 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-kube-api-access-95cvp" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "kube-api-access-95cvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:30.540758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.540724 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-envoy" pod "2b7edc1d-99bc-4df6-93b2-bc945072d2ed" (UID: "2b7edc1d-99bc-4df6-93b2-bc945072d2ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653827 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653863 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653877 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653890 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653903 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653915 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653929 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653944 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.698751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.653960 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95cvp\" (UniqueName: \"kubernetes.io/projected/2b7edc1d-99bc-4df6-93b2-bc945072d2ed-kube-api-access-95cvp\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:30.788601 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.788568 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:30.920780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.915393 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:54:30.920780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.915557 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 12:54:30.920780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:30.919529 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 12:54:31.026585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.026508 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:54:31.026801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.026777 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:54:31.028166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.028137 6727 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:54:31.028268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.028190 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:54:31.034393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.034369 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:54:31.035123 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.035095 6727 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 12:54:31.035210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.035148 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 12:54:31.117484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.116826 6727 generic.go:356] "Generic (PLEG): container finished" podID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerID="3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28" exitCode=0 Oct 31 12:54:31.117484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.116909 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" event={"ID":"2b7edc1d-99bc-4df6-93b2-bc945072d2ed","Type":"ContainerDied","Data":"3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28"} Oct 31 12:54:31.117484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.117022 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" event={"ID":"2b7edc1d-99bc-4df6-93b2-bc945072d2ed","Type":"ContainerDied","Data":"a0b6f983432e9d2b7b13375530be7a96336571c655b0f3ac758f33302fe60b08"} Oct 31 12:54:31.117484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.117337 6727 scope.go:122] "RemoveContainer" containerID="3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28" Oct 31 12:54:31.118271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.118241 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv" Oct 31 12:54:31.131904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.131885 6727 scope.go:122] "RemoveContainer" containerID="3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28" Oct 31 12:54:31.132251 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:31.132229 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28\": container with ID starting with 3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28 not found: ID does not exist" containerID="3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28" Oct 31 12:54:31.132348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.132271 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28"} err="failed to get container status \"3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28\": rpc error: code = NotFound desc = could not find container \"3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28\": container with ID starting with 3f77048577bf6f9c37abe7963b38f93b17eb06d96bbcd6eae97633c18a535c28 not found: ID does not exist" Oct 31 12:54:31.149491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.149458 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv"] Oct 31 12:54:31.154268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.154247 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-z6gkv"] Oct 31 12:54:31.315331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.314288 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" path="/var/lib/kubelet/pods/2b7edc1d-99bc-4df6-93b2-bc945072d2ed/volumes" Oct 31 12:54:31.702485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.690888 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:31.889486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.883635 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:31.904772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.904741 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 12:54:31.905069 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:31.905032 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 12:54:32.034303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.034231 6727 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 12:54:32.034303 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.034278 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 12:54:32.157600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.157570 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:54:32.158947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.158928 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:54:32.670503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.631962 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:32.862158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.862131 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:54:32.934908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.934819 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px"] Oct 31 12:54:32.935668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.935639 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" containerID="cri-o://912bfd35c0aac2f815276b50eaec7247a99d9fd99b9280062315a3e6b94bca58" gracePeriod=30 Oct 31 12:54:32.935995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.934937 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 12:54:32.936166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.936143 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 12:54:32.937014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.936993 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:32.937179 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.937033 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74"] Oct 31 12:54:32.937440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.937421 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" containerID="cri-o://8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361" gracePeriod=30 Oct 31 12:54:32.940388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.940360 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-kube-api-access-gr9s4\" (UniqueName: \"kubernetes.io/projected/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-kube-api-access-gr9s4\") pod \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " Oct 31 12:54:32.941607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.941585 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-plugins-config-volume\") pod \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\" (UID: \"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c\") " Oct 31 12:54:32.943385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.943363 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-plugins-config-volume" pod "6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" (UID: "6f8a5e56-cb16-48ce-afcc-d05ba8efb79c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:33.009160 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:32.982821 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-kube-api-access-gr9s4" pod "6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" (UID: "6f8a5e56-cb16-48ce-afcc-d05ba8efb79c"). InnerVolumeSpecName "kube-api-access-gr9s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:33.039576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.039541 6727 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 12:54:33.039906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.039876 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 12:54:33.106485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.100791 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:33.106485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.100836 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9s4\" (UniqueName: \"kubernetes.io/projected/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c-kube-api-access-gr9s4\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:33.161036 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.161002 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:33.292221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.292136 6727 generic.go:356] "Generic (PLEG): container finished" podID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" containerID="98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9" exitCode=0 Oct 31 12:54:33.294804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.292595 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" Oct 31 12:54:33.295524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.292618 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" event={"ID":"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c","Type":"ContainerDied","Data":"98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9"} Oct 31 12:54:33.295742 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.295723 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr" event={"ID":"6f8a5e56-cb16-48ce-afcc-d05ba8efb79c","Type":"ContainerDied","Data":"8a71b534730e552e0b90bd11e18da6d760e1a9bc7aaed196ad553cc05c841a57"} Oct 31 12:54:33.296638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.296604 6727 scope.go:122] "RemoveContainer" containerID="98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9" Oct 31 12:54:33.348654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.348306 6727 scope.go:122] "RemoveContainer" containerID="98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9" Oct 31 12:54:33.348827 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:33.348770 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9\": container with ID starting with 98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9 not found: ID does not exist" containerID="98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9" Oct 31 12:54:33.348827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.348817 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9"} err="failed to get container status \"98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9\": rpc error: code = NotFound desc = could not find container \"98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9\": container with ID starting with 98e93db4fc35fcffbe64cfa31b62910afc559ca1132e8fdf052f1305f73cf5c9 not found: ID does not exist" Oct 31 12:54:33.432409 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.432378 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr"] Oct 31 12:54:33.436189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.436164 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b218d3-79d6966b67-rhlgr"] Oct 31 12:54:33.913485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.913002 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 12:54:33.913485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.913058 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 12:54:33.936714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:33.936658 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:34.035052 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.035023 6727 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 12:54:34.035224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.035069 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 12:54:34.132044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.131905 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:34.329722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.329679 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:34.346263 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.346228 6727 generic.go:356] "Generic (PLEG): container finished" podID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerID="630d80cb32b11580807edcca86f400263dddbda3907daa79b926e3e3a61b3d5e" exitCode=0 Oct 31 12:54:34.346401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.346336 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" event={"ID":"d9edd6c1-cabc-48e4-98c0-78d0d012af06","Type":"ContainerDied","Data":"630d80cb32b11580807edcca86f400263dddbda3907daa79b926e3e3a61b3d5e"} Oct 31 12:54:34.353225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.352109 6727 generic.go:356] "Generic (PLEG): container finished" podID="a0885b27-54d3-4456-8beb-c8be64f65964" containerID="32931eac714d01b04d5d5e9b3b8c04a14bb27717ec3e216e42b6ab91379fe479" exitCode=0 Oct 31 12:54:34.353225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.352313 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" event={"ID":"a0885b27-54d3-4456-8beb-c8be64f65964","Type":"ContainerDied","Data":"32931eac714d01b04d5d5e9b3b8c04a14bb27717ec3e216e42b6ab91379fe479"} Oct 31 12:54:34.357313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.355689 6727 generic.go:356] "Generic (PLEG): container finished" podID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerID="941f703504ebcb532e4e40b0711e5fb3020834439274682bd4b065775c9dc031" exitCode=0 Oct 31 12:54:34.357313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.355734 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" event={"ID":"f392b872-3c09-4dfc-9fb5-c162bad20d8e","Type":"ContainerDied","Data":"941f703504ebcb532e4e40b0711e5fb3020834439274682bd4b065775c9dc031"} Oct 31 12:54:34.599488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.595737 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:34.723320 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.723212 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756045 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0885b27-54d3-4456-8beb-c8be64f65964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0885b27-54d3-4456-8beb-c8be64f65964-istiod-ca-cert\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756207 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0885b27-54d3-4456-8beb-c8be64f65964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0885b27-54d3-4456-8beb-c8be64f65964-istio-podinfo\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756258 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-socket\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756313 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-certs\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756397 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-credential-socket\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756428 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-istio-token\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-istio-token\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756515 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0885b27-54d3-4456-8beb-c8be64f65964-istiod-ca-cert" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756584 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-kube-api-access-22f8t\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-kube-api-access-22f8t\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756627 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-envoy\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756675 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-data\") pod \"a0885b27-54d3-4456-8beb-c8be64f65964\" (UID: \"a0885b27-54d3-4456-8beb-c8be64f65964\") " Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756780 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-certs" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.756992 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-socket" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.757145 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-credential-socket" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.757479 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.757500 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.757513 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0885b27-54d3-4456-8beb-c8be64f65964-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.757525 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.761354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.760820 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-kube-api-access-22f8t" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "kube-api-access-22f8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:34.762437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.761630 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0885b27-54d3-4456-8beb-c8be64f65964-istio-podinfo" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:34.762437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.761889 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-istio-token" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:34.762437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.761913 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-envoy" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.796493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.780730 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-data" pod "a0885b27-54d3-4456-8beb-c8be64f65964" (UID: "a0885b27-54d3-4456-8beb-c8be64f65964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.853550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.853522 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:54:34.858887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.858866 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-credential-socket\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.858991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.858924 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-data\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.858991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.858955 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istiod-ca-cert\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.858991 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.858979 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-token\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-token\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.859094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859007 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-kube-api-access-p5szp\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-kube-api-access-p5szp\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.859094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859088 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-envoy\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.859158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859109 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-podinfo\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.859158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859138 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-certs\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.859222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859190 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-socket\") pod \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\" (UID: \"f392b872-3c09-4dfc-9fb5-c162bad20d8e\") " Oct 31 12:54:34.859222 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859154 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-credential-socket" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.859297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859205 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-data" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.859657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859535 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:54:34.859657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859547 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istiod-ca-cert" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:34.859849 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859733 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-socket" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.859959 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859939 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859963 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859978 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.859992 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0885b27-54d3-4456-8beb-c8be64f65964-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860017 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.860005 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.860019 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.860033 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22f8t\" (UniqueName: \"kubernetes.io/projected/a0885b27-54d3-4456-8beb-c8be64f65964-kube-api-access-22f8t\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.860050 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.860062 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0885b27-54d3-4456-8beb-c8be64f65964-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.860864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.860842 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-certs" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.861553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.861520 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-kube-api-access-p5szp" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "kube-api-access-p5szp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:34.861760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.861740 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-token" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:34.899497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.876301 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-envoy" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.899497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.876377 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-podinfo" pod "f392b872-3c09-4dfc-9fb5-c162bad20d8e" (UID: "f392b872-3c09-4dfc-9fb5-c162bad20d8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:34.920058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.919655 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 12:54:34.920058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.919709 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 12:54:34.960782 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.960755 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-certs\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961004 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-socket\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961112 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-envoy\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961234 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-data\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961341 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istiod-ca-cert\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961419 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961411 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-kube-api-access-74zch\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-kube-api-access-74zch\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961526 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961517 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-credential-socket\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961689 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961675 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-podinfo\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.961797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961787 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-token\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-token\") pod \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\" (UID: \"d9edd6c1-cabc-48e4-98c0-78d0d012af06\") " Oct 31 12:54:34.962575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961550 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-socket" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.962575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962251 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-data" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.962575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961688 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-certs" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.962575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.961870 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istiod-ca-cert" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:34.962575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962384 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-credential-socket" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962604 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962626 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962641 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962655 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962669 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5szp\" (UniqueName: \"kubernetes.io/projected/f392b872-3c09-4dfc-9fb5-c162bad20d8e-kube-api-access-p5szp\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962683 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962696 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962711 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962725 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f392b872-3c09-4dfc-9fb5-c162bad20d8e-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.962904 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:34.962738 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f392b872-3c09-4dfc-9fb5-c162bad20d8e-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:35.025961 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.025923 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-envoy" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:35.026370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.026331 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-kube-api-access-74zch" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "kube-api-access-74zch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:35.026643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.026598 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-podinfo" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:35.026802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.026784 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-token" pod "d9edd6c1-cabc-48e4-98c0-78d0d012af06" (UID: "d9edd6c1-cabc-48e4-98c0-78d0d012af06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:35.034912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.034717 6727 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 12:54:35.034912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.034780 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 12:54:35.116809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.088305 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:35.116809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.088334 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:35.116809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.088349 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9edd6c1-cabc-48e4-98c0-78d0d012af06-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:35.116809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.088366 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74zch\" (UniqueName: \"kubernetes.io/projected/d9edd6c1-cabc-48e4-98c0-78d0d012af06-kube-api-access-74zch\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:35.324986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.324945 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" path="/var/lib/kubelet/pods/6f8a5e56-cb16-48ce-afcc-d05ba8efb79c/volumes" Oct 31 12:54:35.418935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.418846 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" event={"ID":"d9edd6c1-cabc-48e4-98c0-78d0d012af06","Type":"ContainerDied","Data":"c43515593a9d42eef2a0aa6534c69d3c8d8de2e7a8b7a097ac4b15c50093895a"} Oct 31 12:54:35.418935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.418923 6727 scope.go:122] "RemoveContainer" containerID="630d80cb32b11580807edcca86f400263dddbda3907daa79b926e3e3a61b3d5e" Oct 31 12:54:35.419458 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.419027 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw" Oct 31 12:54:35.424733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.421946 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" event={"ID":"a0885b27-54d3-4456-8beb-c8be64f65964","Type":"ContainerDied","Data":"62c22e6390db0905c9ffb0e60228cb8f1764dd78255f967b760d241e0aa880c6"} Oct 31 12:54:35.424733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.422056 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7" Oct 31 12:54:35.431825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.431773 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" event={"ID":"f392b872-3c09-4dfc-9fb5-c162bad20d8e","Type":"ContainerDied","Data":"36efe3b7ee9b7dc22b2d42cf08e6d9bcbe02591702694383116c181589f1f685"} Oct 31 12:54:35.432322 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.432086 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2" Oct 31 12:54:35.436555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.436519 6727 scope.go:122] "RemoveContainer" containerID="32931eac714d01b04d5d5e9b3b8c04a14bb27717ec3e216e42b6ab91379fe479" Oct 31 12:54:35.451646 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.451529 6727 scope.go:122] "RemoveContainer" containerID="941f703504ebcb532e4e40b0711e5fb3020834439274682bd4b065775c9dc031" Oct 31 12:54:35.456851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.456645 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7"] Oct 31 12:54:35.523738 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.518274 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-lt4g7"] Oct 31 12:54:35.526204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.526175 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw"] Oct 31 12:54:35.530984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.530961 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-hmbnw"] Oct 31 12:54:35.535871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.535831 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2"] Oct 31 12:54:35.539212 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.539179 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-wmmg2"] Oct 31 12:54:35.904488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.901005 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:35.909208 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.908763 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 12:54:35.909208 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:35.908830 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 12:54:36.034177 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:36.034143 6727 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 12:54:36.034366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:36.034202 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 12:54:36.515714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:36.515680 6727 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:54:36.516155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:36.515735 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:54:36.911053 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:36.911014 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:54:36.912404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:36.912379 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:54:36.956660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:36.956626 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:37.039756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:37.039729 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:54:37.041206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:37.041183 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:54:37.303538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:37.303430 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" path="/var/lib/kubelet/pods/a0885b27-54d3-4456-8beb-c8be64f65964/volumes" Oct 31 12:54:37.303969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:37.303952 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" path="/var/lib/kubelet/pods/d9edd6c1-cabc-48e4-98c0-78d0d012af06/volumes" Oct 31 12:54:37.304436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:37.304421 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" path="/var/lib/kubelet/pods/f392b872-3c09-4dfc-9fb5-c162bad20d8e/volumes" Oct 31 12:54:38.057311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.057282 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:38.684756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.684529 6727 generic.go:356] "Generic (PLEG): container finished" podID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerID="912bfd35c0aac2f815276b50eaec7247a99d9fd99b9280062315a3e6b94bca58" exitCode=0 Oct 31 12:54:38.684756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.684630 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" event={"ID":"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3","Type":"ContainerDied","Data":"912bfd35c0aac2f815276b50eaec7247a99d9fd99b9280062315a3e6b94bca58"} Oct 31 12:54:38.851753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.851723 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:54:38.915210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.914047 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:54:38.921875 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.921847 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-token\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-token\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.921880 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-token\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-token\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.921943 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-envoy\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.921983 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-data\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922083 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-credential-socket\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922114 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-socket\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922179 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-kube-api-access-8zqj6\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-kube-api-access-8zqj6\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922210 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-envoy\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922268 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-certs\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922299 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-credential-socket\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922349 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-data\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922376 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ad52d0-9a59-4d88-8a60-90a677715a1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ad52d0-9a59-4d88-8a60-90a677715a1a-istiod-ca-cert\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922426 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-certs\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922486 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istiod-ca-cert\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922536 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-podinfo\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922748 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922676 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-kube-api-access-r767n\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-kube-api-access-r767n\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922742 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-credential-socket" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.922861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922776 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-socket\") pod \"99ad52d0-9a59-4d88-8a60-90a677715a1a\" (UID: \"99ad52d0-9a59-4d88-8a60-90a677715a1a\") " Oct 31 12:54:38.922861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922820 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-podinfo\") pod \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\" (UID: \"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3\") " Oct 31 12:54:38.922978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.922870 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-credential-socket" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.923164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.923139 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-socket" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.923733 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.923702 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-data" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.924326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.924298 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-certs" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.924691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.924665 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istiod-ca-cert" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:38.924783 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.924694 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-envoy" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.925577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925387 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-socket" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.925577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925495 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:38.925577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925515 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:38.925577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925531 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:38.925577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925544 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:38.925577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925557 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:38.925577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925569 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:38.926505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925571 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-data" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.926505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925582 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:38.926505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.925659 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-certs" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.926786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.926595 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ad52d0-9a59-4d88-8a60-90a677715a1a-istiod-ca-cert" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:38.926855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.926817 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-token" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:38.928095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.928060 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-podinfo" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:38.928242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.928225 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-podinfo" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:54:38.928309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.928223 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-envoy" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:54:38.928309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.928237 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-token" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:38.928309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.928256 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-kube-api-access-r767n" pod "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" (UID: "c64ad7a1-e358-452d-8bcf-926b4d4cf2d3"). InnerVolumeSpecName "kube-api-access-r767n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:38.928404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:38.928332 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-kube-api-access-8zqj6" pod "99ad52d0-9a59-4d88-8a60-90a677715a1a" (UID: "99ad52d0-9a59-4d88-8a60-90a677715a1a"). InnerVolumeSpecName "kube-api-access-8zqj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:39.027400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027364 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027394 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r767n\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-kube-api-access-r767n\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027404 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ad52d0-9a59-4d88-8a60-90a677715a1a-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027412 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027422 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027431 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027439 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqj6\" (UniqueName: \"kubernetes.io/projected/99ad52d0-9a59-4d88-8a60-90a677715a1a-kube-api-access-8zqj6\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027447 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027454 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027501 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ad52d0-9a59-4d88-8a60-90a677715a1a-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.027637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.027512 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:39.075110 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.075021 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:39.480371 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.480334 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:39.623060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.622975 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:39.695030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.694991 6727 generic.go:356] "Generic (PLEG): container finished" podID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerID="8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361" exitCode=0 Oct 31 12:54:39.695230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.695070 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" event={"ID":"99ad52d0-9a59-4d88-8a60-90a677715a1a","Type":"ContainerDied","Data":"8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361"} Oct 31 12:54:39.695230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.695077 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" Oct 31 12:54:39.695230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.695157 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74" event={"ID":"99ad52d0-9a59-4d88-8a60-90a677715a1a","Type":"ContainerDied","Data":"a6d19aa7a27cfc4dbe7fc0cbc53a0f22973ab6ad2b48a27bc2b0698f8b3ffda2"} Oct 31 12:54:39.695230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.695190 6727 scope.go:122] "RemoveContainer" containerID="8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361" Oct 31 12:54:39.697013 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.696990 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" Oct 31 12:54:39.697120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.697024 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px" event={"ID":"c64ad7a1-e358-452d-8bcf-926b4d4cf2d3","Type":"ContainerDied","Data":"3d30067f7623ea2fe8d2f2dcb3f91fd799961293bf5971da6f8ca5141d5c9aaf"} Oct 31 12:54:39.710546 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.710523 6727 scope.go:122] "RemoveContainer" containerID="8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361" Oct 31 12:54:39.710920 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:54:39.710901 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361\": container with ID starting with 8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361 not found: ID does not exist" containerID="8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361" Oct 31 12:54:39.710971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.710929 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361"} err="failed to get container status \"8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361\": rpc error: code = NotFound desc = could not find container \"8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361\": container with ID starting with 8faa7091cc41ed1bcfd92ff6cfbd2f2e62fe3804bba3985ae339e4de67612361 not found: ID does not exist" Oct 31 12:54:39.710971 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.710947 6727 scope.go:122] "RemoveContainer" containerID="912bfd35c0aac2f815276b50eaec7247a99d9fd99b9280062315a3e6b94bca58" Oct 31 12:54:39.737368 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.737341 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74"] Oct 31 12:54:39.742513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.742445 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-s6h74"] Oct 31 12:54:39.765217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.765186 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px"] Oct 31 12:54:39.771537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.770779 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xw4px"] Oct 31 12:54:39.834757 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:39.834722 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:40.086004 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:40.085943 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:40.241906 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:40.241866 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:40.404967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:40.404876 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:40.628914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:40.628881 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:40.851606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:40.851564 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:41.054702 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:41.054675 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:41.256156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:41.256065 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:41.302835 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:41.302807 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" path="/var/lib/kubelet/pods/99ad52d0-9a59-4d88-8a60-90a677715a1a/volumes" Oct 31 12:54:41.303348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:41.303332 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" path="/var/lib/kubelet/pods/c64ad7a1-e358-452d-8bcf-926b4d4cf2d3/volumes" Oct 31 12:54:42.380743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:42.380702 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:43.263293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:43.263235 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:43.644943 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:43.644915 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2"] Oct 31 12:54:43.741776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:43.741746 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:43.944571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:43.944422 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:44.154487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:44.148618 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:44.925083 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:44.925001 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:45.629245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:45.629196 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:46.355446 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:46.355407 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:47.122335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:47.122292 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:47.365492 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:47.365450 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:47.506287 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:47.506210 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:47.779095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:47.778993 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:47.948788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:47.948757 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:48.645548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:48.645513 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" podUID="93c56ad8-7acd-496c-9b0d-c9569405e026" containerName="payload-pre-processing" containerID="cri-o://26bf36c22b93e0b94dab1ca55482dbfe10b2e062a890cf7b54cdc72b877eecc4" gracePeriod=25 Oct 31 12:54:48.773293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:48.773262 6727 generic.go:356] "Generic (PLEG): container finished" podID="93c56ad8-7acd-496c-9b0d-c9569405e026" containerID="26bf36c22b93e0b94dab1ca55482dbfe10b2e062a890cf7b54cdc72b877eecc4" exitCode=0 Oct 31 12:54:48.773423 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:48.773328 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" event={"ID":"93c56ad8-7acd-496c-9b0d-c9569405e026","Type":"ContainerDied","Data":"26bf36c22b93e0b94dab1ca55482dbfe10b2e062a890cf7b54cdc72b877eecc4"} Oct 31 12:54:48.928329 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:48.928304 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:54:48.966403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:48.966364 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:49.121274 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.121246 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c56ad8-7acd-496c-9b0d-c9569405e026-kube-api-access-m8w7n\" (UniqueName: \"kubernetes.io/projected/93c56ad8-7acd-496c-9b0d-c9569405e026-kube-api-access-m8w7n\") pod \"93c56ad8-7acd-496c-9b0d-c9569405e026\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " Oct 31 12:54:49.121533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.121512 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c56ad8-7acd-496c-9b0d-c9569405e026-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c56ad8-7acd-496c-9b0d-c9569405e026-plugins-config-volume\") pod \"93c56ad8-7acd-496c-9b0d-c9569405e026\" (UID: \"93c56ad8-7acd-496c-9b0d-c9569405e026\") " Oct 31 12:54:49.121944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.121918 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c56ad8-7acd-496c-9b0d-c9569405e026-plugins-config-volume" pod "93c56ad8-7acd-496c-9b0d-c9569405e026" (UID: "93c56ad8-7acd-496c-9b0d-c9569405e026"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:49.122293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.122277 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c56ad8-7acd-496c-9b0d-c9569405e026-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:49.123229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.123204 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c56ad8-7acd-496c-9b0d-c9569405e026-kube-api-access-m8w7n" pod "93c56ad8-7acd-496c-9b0d-c9569405e026" (UID: "93c56ad8-7acd-496c-9b0d-c9569405e026"). InnerVolumeSpecName "kube-api-access-m8w7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:49.223965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.223904 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8w7n\" (UniqueName: \"kubernetes.io/projected/93c56ad8-7acd-496c-9b0d-c9569405e026-kube-api-access-m8w7n\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:49.549020 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.548943 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:49.734968 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.734932 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:49.786089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.786037 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" event={"ID":"93c56ad8-7acd-496c-9b0d-c9569405e026","Type":"ContainerDied","Data":"dfaf9dd5450caba80741265326464ed2497c46f7123d0f24472c2fe991d8e7ee"} Oct 31 12:54:49.786290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.786118 6727 scope.go:122] "RemoveContainer" containerID="26bf36c22b93e0b94dab1ca55482dbfe10b2e062a890cf7b54cdc72b877eecc4" Oct 31 12:54:49.786290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.786281 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2" Oct 31 12:54:49.807620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.807593 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2"] Oct 31 12:54:49.812435 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.812406 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d4dd68-5ddcd54b59-b2jh2"] Oct 31 12:54:49.876009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:49.875971 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:50.332716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:50.332683 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:50.550718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:50.550676 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl"] Oct 31 12:54:50.551174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:50.550999 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" podUID="64bea21e-91b0-4b3d-9a54-1f147f79e546" containerName="maas-api" containerID="cri-o://831d7ba880945af08b0e038bb2628721a350f3fe16f4e9a8fddc1aaea9b50a81" gracePeriod=30 Oct 31 12:54:50.646573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:50.646543 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:51.013221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.013136 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:51.040965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.040935 6727 generic.go:356] "Generic (PLEG): container finished" podID="64bea21e-91b0-4b3d-9a54-1f147f79e546" containerID="831d7ba880945af08b0e038bb2628721a350f3fe16f4e9a8fddc1aaea9b50a81" exitCode=0 Oct 31 12:54:51.041112 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.041007 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" event={"ID":"64bea21e-91b0-4b3d-9a54-1f147f79e546","Type":"ContainerDied","Data":"831d7ba880945af08b0e038bb2628721a350f3fe16f4e9a8fddc1aaea9b50a81"} Oct 31 12:54:51.073829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.073796 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:54:51.211887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.211856 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bea21e-91b0-4b3d-9a54-1f147f79e546-kube-api-access-q52fm\" (UniqueName: \"kubernetes.io/projected/64bea21e-91b0-4b3d-9a54-1f147f79e546-kube-api-access-q52fm\") pod \"64bea21e-91b0-4b3d-9a54-1f147f79e546\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " Oct 31 12:54:51.212076 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.211999 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-maas-api-tls\") pod \"64bea21e-91b0-4b3d-9a54-1f147f79e546\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " Oct 31 12:54:51.212261 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.212240 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-metrics-certs\") pod \"64bea21e-91b0-4b3d-9a54-1f147f79e546\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " Oct 31 12:54:51.212342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.212329 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bea21e-91b0-4b3d-9a54-1f147f79e546-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64bea21e-91b0-4b3d-9a54-1f147f79e546-openshift-service-ca\") pod \"64bea21e-91b0-4b3d-9a54-1f147f79e546\" (UID: \"64bea21e-91b0-4b3d-9a54-1f147f79e546\") " Oct 31 12:54:51.212883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.212845 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bea21e-91b0-4b3d-9a54-1f147f79e546-openshift-service-ca" pod "64bea21e-91b0-4b3d-9a54-1f147f79e546" (UID: "64bea21e-91b0-4b3d-9a54-1f147f79e546"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:51.213029 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.213012 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64bea21e-91b0-4b3d-9a54-1f147f79e546-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:51.214883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.214855 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-metrics-certs" pod "64bea21e-91b0-4b3d-9a54-1f147f79e546" (UID: "64bea21e-91b0-4b3d-9a54-1f147f79e546"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:51.214983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.214906 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-maas-api-tls" pod "64bea21e-91b0-4b3d-9a54-1f147f79e546" (UID: "64bea21e-91b0-4b3d-9a54-1f147f79e546"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:51.215032 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.214985 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bea21e-91b0-4b3d-9a54-1f147f79e546-kube-api-access-q52fm" pod "64bea21e-91b0-4b3d-9a54-1f147f79e546" (UID: "64bea21e-91b0-4b3d-9a54-1f147f79e546"). InnerVolumeSpecName "kube-api-access-q52fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:51.240424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.240396 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:51.283879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.283815 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:54:51.302905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.302877 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c56ad8-7acd-496c-9b0d-c9569405e026" path="/var/lib/kubelet/pods/93c56ad8-7acd-496c-9b0d-c9569405e026/volumes" Oct 31 12:54:51.315148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.315131 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:51.315148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.315150 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q52fm\" (UniqueName: \"kubernetes.io/projected/64bea21e-91b0-4b3d-9a54-1f147f79e546-kube-api-access-q52fm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:51.315305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.315165 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64bea21e-91b0-4b3d-9a54-1f147f79e546-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:51.491337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.489451 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:51.631774 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:51.631635 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:52.050618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.050535 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" Oct 31 12:54:52.050618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.050552 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl" event={"ID":"64bea21e-91b0-4b3d-9a54-1f147f79e546","Type":"ContainerDied","Data":"833253b22b961e39f0214a5d0b6afa074c7bb07db7d484d560207bf0309aa451"} Oct 31 12:54:52.051105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.050629 6727 scope.go:122] "RemoveContainer" containerID="831d7ba880945af08b0e038bb2628721a350f3fe16f4e9a8fddc1aaea9b50a81" Oct 31 12:54:52.072564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.072543 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl"] Oct 31 12:54:52.078223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.078195 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d4dd68-745597fd7c-nj2zl"] Oct 31 12:54:52.226859 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.226826 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:52.958751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.958722 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt"] Oct 31 12:54:52.959581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959564 6727 cpu_manager.go:414] "containerMap: removing container" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" Oct 31 12:54:52.959662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959584 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" Oct 31 12:54:52.959662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959599 6727 cpu_manager.go:414] "containerMap: removing container" podUID="64bea21e-91b0-4b3d-9a54-1f147f79e546" containerName="maas-api" Oct 31 12:54:52.959662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959606 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bea21e-91b0-4b3d-9a54-1f147f79e546" containerName="maas-api" Oct 31 12:54:52.959662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959634 6727 cpu_manager.go:414] "containerMap: removing container" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" Oct 31 12:54:52.959662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959643 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" Oct 31 12:54:52.959662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959656 6727 cpu_manager.go:414] "containerMap: removing container" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" Oct 31 12:54:52.959662 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959663 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959674 6727 cpu_manager.go:414] "containerMap: removing container" podUID="9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" containerName="maas-api" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959680 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" containerName="maas-api" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959694 6727 cpu_manager.go:414] "containerMap: removing container" podUID="46f3751c-f307-4903-ab02-f236dc860401" containerName="payload-pre-processing" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959701 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f3751c-f307-4903-ab02-f236dc860401" containerName="payload-pre-processing" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959728 6727 cpu_manager.go:414] "containerMap: removing container" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959736 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959750 6727 cpu_manager.go:414] "containerMap: removing container" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959756 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959769 6727 cpu_manager.go:414] "containerMap: removing container" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959775 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959788 6727 cpu_manager.go:414] "containerMap: removing container" podUID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" containerName="payload-pre-processing" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959795 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" containerName="payload-pre-processing" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959804 6727 cpu_manager.go:414] "containerMap: removing container" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959810 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959822 6727 cpu_manager.go:414] "containerMap: removing container" podUID="93c56ad8-7acd-496c-9b0d-c9569405e026" containerName="payload-pre-processing" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959829 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c56ad8-7acd-496c-9b0d-c9569405e026" containerName="payload-pre-processing" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959849 6727 cpu_manager.go:414] "containerMap: removing container" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959855 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959869 6727 cpu_manager.go:414] "containerMap: removing container" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959876 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959889 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959895 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959914 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" Oct 31 12:54:52.960024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.959921 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960107 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="06585f3d-576a-4abd-94b5-1ae90777820b" containerName="main" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960123 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bea21e-91b0-4b3d-9a54-1f147f79e546" containerName="maas-api" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960136 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9edd6c1-cabc-48e4-98c0-78d0d012af06" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960147 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="69da0e2c-7478-499f-879e-20aa90d6ad89" containerName="main" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960159 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8dd1bb-ee38-4ebb-a594-ccbbef643fa3" containerName="maas-api" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960168 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f3751c-f307-4903-ab02-f236dc860401" containerName="payload-pre-processing" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960180 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0885b27-54d3-4456-8beb-c8be64f65964" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960190 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ad52d0-9a59-4d88-8a60-90a677715a1a" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960200 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7edc1d-99bc-4df6-93b2-bc945072d2ed" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960208 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8a5e56-cb16-48ce-afcc-d05ba8efb79c" containerName="payload-pre-processing" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960217 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d09a8c-25dd-4644-8455-0e9c5ebed221" containerName="main" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960226 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c56ad8-7acd-496c-9b0d-c9569405e026" containerName="payload-pre-processing" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960236 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="771d25b0-f843-4e87-8424-cfbb691f3173" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960243 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ad7a1-e358-452d-8bcf-926b4d4cf2d3" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960250 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57f48ec-1e88-4fb4-9569-7443b23107fa" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960259 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f392b872-3c09-4dfc-9fb5-c162bad20d8e" containerName="istio-proxy" Oct 31 12:54:52.960856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.960268 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0463c520-2269-4666-a30c-e6f36b31f750" containerName="istio-proxy" Oct 31 12:54:52.965574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.965556 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:52.981143 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.981121 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b62a0\"/\"test-model-9b62a0-kserve-self-signed-certs\"" Oct 31 12:54:52.984691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.984671 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt"] Oct 31 12:54:52.989010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:52.988988 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b62a0\"/\"default-dockercfg-stj7r\"" Oct 31 12:54:53.000343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.000322 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b62a0\"/\"kube-root-ca.crt\"" Oct 31 12:54:53.000343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.000326 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b62a0\"/\"openshift-service-ca.crt\"" Oct 31 12:54:53.051440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.051415 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-model-cache\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.051885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.051450 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tls-certs\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.051885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.051697 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tmp-dir\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.051885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.051746 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-dshm\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.051885 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.051877 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdww\" (UniqueName: \"kubernetes.io/projected/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-kube-api-access-vrdww\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.052034 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.051936 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-home\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.081459 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.081435 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:53.153102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153074 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tmp-dir\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153111 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-dshm\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153152 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdww\" (UniqueName: \"kubernetes.io/projected/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-kube-api-access-vrdww\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153174 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-home\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153307 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-model-cache\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153335 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tls-certs\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153514 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tmp-dir\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153665 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153639 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-model-cache\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.153703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.153652 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-home\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.155587 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.155560 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-dshm\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.155739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.155722 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tls-certs\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.178529 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.178505 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdww\" (UniqueName: \"kubernetes.io/projected/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-kube-api-access-vrdww\") pod \"test-model-9b62a0-kserve-7698b68fb7-28ljt\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.280540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.279567 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:53.428525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.427693 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bea21e-91b0-4b3d-9a54-1f147f79e546" path="/var/lib/kubelet/pods/64bea21e-91b0-4b3d-9a54-1f147f79e546/volumes" Oct 31 12:54:53.849393 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.840388 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt"] Oct 31 12:54:53.889934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:53.889088 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:54.085535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:54.077955 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" event={"ID":"c60a1c48-7c98-4f57-bda4-1d5aa6be1273","Type":"ContainerStarted","Data":"5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d"} Oct 31 12:54:54.085535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:54.078010 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" event={"ID":"c60a1c48-7c98-4f57-bda4-1d5aa6be1273","Type":"ContainerStarted","Data":"5a07ea8c28fa35ef90886eaf1b2d493c615ae6f5df2d56283411b014572cefb8"} Oct 31 12:54:54.113438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:54.110603 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" podStartSLOduration=2.11058538 podStartE2EDuration="2.11058538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:54.099383459 +0000 UTC m=+1604.914397851" watchObservedRunningTime="2026-10-31 12:54:54.11058538 +0000 UTC m=+1604.925599775" Oct 31 12:54:54.285307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:54.285262 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:54:54.831673 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:54.831643 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7"] Oct 31 12:54:54.831876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:54.831851 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" containerID="cri-o://fad0c3791b26a357167106109b39ce240a3db6f2da4fa72e0058d7dd0aa948f6" gracePeriod=30 Oct 31 12:54:54.912058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:54.912029 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:55.128537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:55.128420 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4"] Oct 31 12:54:55.128915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:55.128610 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" containerID="cri-o://3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd" gracePeriod=30 Oct 31 12:54:55.387108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:55.387033 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:56.215714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:56.215683 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:57.036012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:57.035957 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:57.732884 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:57.732838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:58.315996 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:58.315965 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:58.700736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:58.700700 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:59.505029 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.504995 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:54:59.505424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.505074 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:54:59.596181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.596146 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:59.768945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.768876 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:59.889049 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.889017 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs"] Oct 31 12:54:59.889289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.889262 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" containerID="cri-o://1bbacec97c73367ecde0a2a7182859f46480d1c5bdd79b37113318e2309f66b6" gracePeriod=30 Oct 31 12:54:59.894897 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.894871 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c"] Oct 31 12:54:59.895137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.895103 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" containerID="cri-o://5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e" gracePeriod=30 Oct 31 12:54:59.933633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:54:59.933603 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:00.092063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.092030 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:00.167552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.164617 6727 generic.go:356] "Generic (PLEG): container finished" podID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerID="fad0c3791b26a357167106109b39ce240a3db6f2da4fa72e0058d7dd0aa948f6" exitCode=0 Oct 31 12:55:00.167552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.164800 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" event={"ID":"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef","Type":"ContainerDied","Data":"fad0c3791b26a357167106109b39ce240a3db6f2da4fa72e0058d7dd0aa948f6"} Oct 31 12:55:00.167552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.164849 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" event={"ID":"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef","Type":"ContainerDied","Data":"d644c21a4644a0f5fde096ade07ec7caee3ebea6dd8bf17a18ee5806b7580b70"} Oct 31 12:55:00.167552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.164871 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d644c21a4644a0f5fde096ade07ec7caee3ebea6dd8bf17a18ee5806b7580b70" Oct 31 12:55:00.235811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.231988 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:55:00.336559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.309049 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:00.359385 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:00.359312 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.363375 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-certs\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.363505 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-token\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-token\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.363546 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-data\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.363589 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-credential-socket\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.363633 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-socket\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.363733 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istiod-ca-cert\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.364007 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-envoy\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.364106 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-podinfo\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.366623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.364226 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-kube-api-access-chx4p\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-kube-api-access-chx4p\") pod \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\" (UID: \"3bc8ebb8-d3bf-4c04-a793-25e2810b70ef\") " Oct 31 12:55:00.375531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.368048 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istiod-ca-cert" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:00.375531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.368349 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-socket" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.375531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.368678 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-data" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.375531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.368737 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-certs" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.375531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.368784 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-credential-socket" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.375531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.371962 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-podinfo" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:55:00.385667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.384076 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-kube-api-access-chx4p" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "kube-api-access-chx4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:00.385667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.384196 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-token" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:00.385667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.384583 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-envoy" pod "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" (UID: "3bc8ebb8-d3bf-4c04-a793-25e2810b70ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.470888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470854 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.470888 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470890 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.471119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470905 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.471119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470919 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.471119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470934 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.471119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470947 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.471119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470960 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.471119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470975 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chx4p\" (UniqueName: \"kubernetes.io/projected/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-kube-api-access-chx4p\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.471119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.470991 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.472669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.472645 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:55:00.572236 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.572202 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-credential-socket\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.572731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.572306 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-socket\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.572731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.572329 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-istio-token\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-istio-token\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.572731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.572350 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-data\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.572731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.572421 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-kube-api-access-xzfjf\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-kube-api-access-xzfjf\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.572731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.572587 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-credential-socket" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.572731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.572709 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-data" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.573042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.573018 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20154455-637d-4431-a426-54d6ad76f818-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20154455-637d-4431-a426-54d6ad76f818-istiod-ca-cert\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.573118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.573081 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20154455-637d-4431-a426-54d6ad76f818-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20154455-637d-4431-a426-54d6ad76f818-istio-podinfo\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.573188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.573171 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-certs\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.573294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.573279 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-envoy\") pod \"20154455-637d-4431-a426-54d6ad76f818\" (UID: \"20154455-637d-4431-a426-54d6ad76f818\") " Oct 31 12:55:00.573680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.573655 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20154455-637d-4431-a426-54d6ad76f818-istiod-ca-cert" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:00.573810 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.573725 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-socket" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.574012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.573977 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-certs" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.575016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.574990 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-istio-token" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:00.575573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.575551 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.575652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.575580 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.575652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.575595 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.575652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.575609 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20154455-637d-4431-a426-54d6ad76f818-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.575776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.575625 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.575776 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.575734 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-kube-api-access-xzfjf" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "kube-api-access-xzfjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:00.576363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.576338 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-envoy" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:00.576838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.576814 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20154455-637d-4431-a426-54d6ad76f818-istio-podinfo" pod "20154455-637d-4431-a426-54d6ad76f818" (UID: "20154455-637d-4431-a426-54d6ad76f818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:55:00.603767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.603683 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq"] Oct 31 12:55:00.604373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.604338 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" containerID="cri-o://b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc" gracePeriod=30 Oct 31 12:55:00.608562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.608531 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:55:00.608703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.608593 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:55:00.677149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.677110 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.677149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.677150 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzfjf\" (UniqueName: \"kubernetes.io/projected/20154455-637d-4431-a426-54d6ad76f818-kube-api-access-xzfjf\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.677634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.677165 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20154455-637d-4431-a426-54d6ad76f818-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.677634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.677178 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20154455-637d-4431-a426-54d6ad76f818-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:00.742486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:00.741649 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:01.006928 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.006897 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:01.174493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.174384 6727 generic.go:356] "Generic (PLEG): container finished" podID="20154455-637d-4431-a426-54d6ad76f818" containerID="3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd" exitCode=0 Oct 31 12:55:01.174493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.174440 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" Oct 31 12:55:01.174493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.174456 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" event={"ID":"20154455-637d-4431-a426-54d6ad76f818","Type":"ContainerDied","Data":"3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd"} Oct 31 12:55:01.174759 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.174525 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4" event={"ID":"20154455-637d-4431-a426-54d6ad76f818","Type":"ContainerDied","Data":"5376df27f2568c9daa2cab808d8af53fcb1e202e3f89924738b94656d5db037f"} Oct 31 12:55:01.174759 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.174551 6727 scope.go:122] "RemoveContainer" containerID="3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd" Oct 31 12:55:01.174759 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.174695 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7" Oct 31 12:55:01.188206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.188180 6727 scope.go:122] "RemoveContainer" containerID="3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd" Oct 31 12:55:01.188554 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:01.188528 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd\": container with ID starting with 3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd not found: ID does not exist" containerID="3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd" Oct 31 12:55:01.188620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.188564 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd"} err="failed to get container status \"3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd\": rpc error: code = NotFound desc = could not find container \"3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd\": container with ID starting with 3951adea04c19d58cb3a85a0d72d1a486897ef4913f4ae39136f21ef984924cd not found: ID does not exist" Oct 31 12:55:01.212864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.212832 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7"] Oct 31 12:55:01.222009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.221983 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-2jxj7"] Oct 31 12:55:01.239132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.239109 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4"] Oct 31 12:55:01.239827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.239807 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:01.247117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.247097 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-mr4m4"] Oct 31 12:55:01.307346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.307317 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20154455-637d-4431-a426-54d6ad76f818" path="/var/lib/kubelet/pods/20154455-637d-4431-a426-54d6ad76f818/volumes" Oct 31 12:55:01.308059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:01.308020 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" path="/var/lib/kubelet/pods/3bc8ebb8-d3bf-4c04-a793-25e2810b70ef/volumes" Oct 31 12:55:02.003889 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.003854 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:02.005447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.005424 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb"] Oct 31 12:55:02.006275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.006257 6727 cpu_manager.go:414] "containerMap: removing container" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" Oct 31 12:55:02.006275 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.006275 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" Oct 31 12:55:02.006391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.006310 6727 cpu_manager.go:414] "containerMap: removing container" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" Oct 31 12:55:02.006391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.006315 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" Oct 31 12:55:02.006491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.006416 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc8ebb8-d3bf-4c04-a793-25e2810b70ef" containerName="istio-proxy" Oct 31 12:55:02.006491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.006432 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="20154455-637d-4431-a426-54d6ad76f818" containerName="istio-proxy" Oct 31 12:55:02.011491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.011442 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.021754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.021062 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb"] Oct 31 12:55:02.022439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.022420 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d086f3\"/\"openshift-service-ca.crt\"" Oct 31 12:55:02.022982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.022959 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d086f3\"/\"test-model-d086f3-kserve-self-signed-certs\"" Oct 31 12:55:02.023064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.022966 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d086f3\"/\"kube-root-ca.crt\"" Oct 31 12:55:02.023064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.023015 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d086f3\"/\"default-dockercfg-gnjps\"" Oct 31 12:55:02.099535 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.099503 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-dshm\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.099654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.099543 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tmp-dir\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.099696 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.099658 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tls-certs\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.099786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.099768 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-home\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.099847 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.099835 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vbc\" (UniqueName: \"kubernetes.io/projected/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-kube-api-access-72vbc\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.099917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.099906 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-model-cache\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.163118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.163094 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:02.210486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.207541 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-dshm\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.210486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.207602 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tmp-dir\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.210486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.207697 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tls-certs\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.210486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.207829 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-home\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.210486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.207926 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vbc\" (UniqueName: \"kubernetes.io/projected/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-kube-api-access-72vbc\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.210486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.207996 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-model-cache\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.210486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.208455 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-model-cache\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.212675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.212310 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tls-certs\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.212675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.212631 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tmp-dir\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.215858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.215835 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-home\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.219995 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.219866 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-dshm\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.228310 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.228029 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vbc\" (UniqueName: \"kubernetes.io/projected/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-kube-api-access-72vbc\") pod \"test-model-d086f3-kserve-675466565-lw8qb\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.242967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.238018 6727 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:55:02.242967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.238104 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:55:02.328210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.328182 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:02.618453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.618425 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb"] Oct 31 12:55:02.619016 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:55:02.618993 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888b8db5_5f7a_49f1_91a7_7ddabdd54f00.slice/crio-16c13008476331419de1fa4a5a11d43f26103cf700e0fcdf39c2522ac990bf9f WatchSource:0}: Error finding container 16c13008476331419de1fa4a5a11d43f26103cf700e0fcdf39c2522ac990bf9f: Status 404 returned error can't find the container with id 16c13008476331419de1fa4a5a11d43f26103cf700e0fcdf39c2522ac990bf9f Oct 31 12:55:02.773857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:02.773823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:03.230131 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:03.230096 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" event={"ID":"888b8db5-5f7a-49f1-91a7-7ddabdd54f00","Type":"ContainerStarted","Data":"cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4"} Oct 31 12:55:03.230508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:03.230146 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" event={"ID":"888b8db5-5f7a-49f1-91a7-7ddabdd54f00","Type":"ContainerStarted","Data":"16c13008476331419de1fa4a5a11d43f26103cf700e0fcdf39c2522ac990bf9f"} Oct 31 12:55:03.255976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:03.255931 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" podStartSLOduration=2.25591761 podStartE2EDuration="2.25591761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:03.250927027 +0000 UTC m=+1614.065941417" watchObservedRunningTime="2026-10-31 12:55:03.25591761 +0000 UTC m=+1614.070931999" Oct 31 12:55:03.280578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:03.280554 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:55:03.301631 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:03.301609 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:55:03.328448 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:03.328425 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:03.621970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:03.621939 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:04.249162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:04.249137 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:55:04.361358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:04.361323 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:05.150717 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.149716 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:05.215511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.215452 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:55:05.251319 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.251268 6727 generic.go:356] "Generic (PLEG): container finished" podID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerID="5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e" exitCode=0 Oct 31 12:55:05.251712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.251377 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" Oct 31 12:55:05.251712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.251376 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" event={"ID":"89263210-c66c-4aea-8a82-0849d93f0a7a","Type":"ContainerDied","Data":"5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e"} Oct 31 12:55:05.251712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.251562 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c" event={"ID":"89263210-c66c-4aea-8a82-0849d93f0a7a","Type":"ContainerDied","Data":"93fc917a90cfa0c4d2f52363584bc84727b7b5f5e6dd2daabf6bb4d024f35b61"} Oct 31 12:55:05.251712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.251602 6727 scope.go:122] "RemoveContainer" containerID="5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e" Oct 31 12:55:05.253064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.253041 6727 generic.go:356] "Generic (PLEG): container finished" podID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerID="1bbacec97c73367ecde0a2a7182859f46480d1c5bdd79b37113318e2309f66b6" exitCode=0 Oct 31 12:55:05.253174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.253109 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" event={"ID":"aafa41b6-0956-45e8-8b12-481bbbcaa965","Type":"ContainerDied","Data":"1bbacec97c73367ecde0a2a7182859f46480d1c5bdd79b37113318e2309f66b6"} Oct 31 12:55:05.253174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.253157 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" event={"ID":"aafa41b6-0956-45e8-8b12-481bbbcaa965","Type":"ContainerDied","Data":"e776c9c553801e99c06e56692d2e84defd550d039d34b5ac6a4914bfc785adc0"} Oct 31 12:55:05.253273 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.253180 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e776c9c553801e99c06e56692d2e84defd550d039d34b5ac6a4914bfc785adc0" Oct 31 12:55:05.257067 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.257050 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:55:05.264103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.264072 6727 scope.go:122] "RemoveContainer" containerID="5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e" Oct 31 12:55:05.264407 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:05.264376 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e\": container with ID starting with 5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e not found: ID does not exist" containerID="5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e" Oct 31 12:55:05.264545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.264406 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e"} err="failed to get container status \"5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e\": rpc error: code = NotFound desc = could not find container \"5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e\": container with ID starting with 5f844151595da29a74e1bb74e3f868ea12d94f2eb61f8d32901cfe24c778c48e not found: ID does not exist" Oct 31 12:55:05.360383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.360355 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-envoy\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.360680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.360654 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-podinfo\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.360680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.360693 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-podinfo\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.361000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.360780 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-socket\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.360815 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-kube-api-access-jm79h\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-kube-api-access-jm79h\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.361000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.360879 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-socket\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.361000 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.360961 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-certs\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361022 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-data\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361054 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-token\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361103 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-credential-socket\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.361231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361190 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafa41b6-0956-45e8-8b12-481bbbcaa965-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafa41b6-0956-45e8-8b12-481bbbcaa965-istiod-ca-cert\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.361231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361226 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-credential-socket\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361360 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-data\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.361570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361434 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-envoy\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361483 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89263210-c66c-4aea-8a82-0849d93f0a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89263210-c66c-4aea-8a82-0849d93f0a7a-istiod-ca-cert\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361566 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-certs\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.361770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361609 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-kube-api-access-bvshm\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-kube-api-access-bvshm\") pod \"89263210-c66c-4aea-8a82-0849d93f0a7a\" (UID: \"89263210-c66c-4aea-8a82-0849d93f0a7a\") " Oct 31 12:55:05.361770 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.361656 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-token\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-token\") pod \"aafa41b6-0956-45e8-8b12-481bbbcaa965\" (UID: \"aafa41b6-0956-45e8-8b12-481bbbcaa965\") " Oct 31 12:55:05.362174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.362150 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-credential-socket" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.362822 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.362798 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.362941 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.362857 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-socket" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.363369 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.363343 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-socket" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.363641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.363119 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafa41b6-0956-45e8-8b12-481bbbcaa965-istiod-ca-cert" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:05.363729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.363656 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89263210-c66c-4aea-8a82-0849d93f0a7a-istiod-ca-cert" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:05.364427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.363941 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-certs" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.364427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.363980 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-credential-socket" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.364427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.364016 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-data" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.364799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.364777 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-certs" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.365137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.365117 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-data" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.367291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.367257 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-envoy" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.369030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.369009 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-kube-api-access-jm79h" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "kube-api-access-jm79h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:05.369638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.369607 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-token" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:05.369842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.369807 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-podinfo" pod "aafa41b6-0956-45e8-8b12-481bbbcaa965" (UID: "aafa41b6-0956-45e8-8b12-481bbbcaa965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:55:05.369929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.369864 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-podinfo" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:55:05.370133 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.370104 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-token" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:05.370954 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.370922 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-envoy" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:05.371237 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.371214 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-kube-api-access-bvshm" pod "89263210-c66c-4aea-8a82-0849d93f0a7a" (UID: "89263210-c66c-4aea-8a82-0849d93f0a7a"). InnerVolumeSpecName "kube-api-access-bvshm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:05.464414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464352 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464374 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464385 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464397 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464411 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm79h\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-kube-api-access-jm79h\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464421 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464430 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464438 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464446 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464455 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafa41b6-0956-45e8-8b12-481bbbcaa965-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464499 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464511 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464518 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89263210-c66c-4aea-8a82-0849d93f0a7a-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464526 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89263210-c66c-4aea-8a82-0849d93f0a7a-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464534 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafa41b6-0956-45e8-8b12-481bbbcaa965-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464543 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvshm\" (UniqueName: \"kubernetes.io/projected/89263210-c66c-4aea-8a82-0849d93f0a7a-kube-api-access-bvshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.464732 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.464551 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafa41b6-0956-45e8-8b12-481bbbcaa965-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:05.591394 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.591361 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c"] Oct 31 12:55:05.595210 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.595187 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-grz9c"] Oct 31 12:55:05.911021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:05.910995 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:55:06.042808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.042720 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8"] Oct 31 12:55:06.071148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071110 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f585440-9878-4540-b806-6704375dc496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f585440-9878-4540-b806-6704375dc496-istiod-ca-cert\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071204 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-credential-socket\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071306 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-data\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071451 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071414 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f585440-9878-4540-b806-6704375dc496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f585440-9878-4540-b806-6704375dc496-istio-podinfo\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071536 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-kube-api-access-5cq88\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-kube-api-access-5cq88\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071588 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-socket\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071638 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071623 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-istio-token\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-istio-token\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071679 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071670 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-envoy\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.071760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071748 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-certs\") pod \"7f585440-9878-4540-b806-6704375dc496\" (UID: \"7f585440-9878-4540-b806-6704375dc496\") " Oct 31 12:55:06.072308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071590 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-credential-socket" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:06.072308 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.071738 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f585440-9878-4540-b806-6704375dc496-istiod-ca-cert" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:06.072788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.072741 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-certs" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:06.072788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.072777 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-socket" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:06.072951 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.072822 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-data" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:06.073552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.073529 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f585440-9878-4540-b806-6704375dc496-istio-podinfo" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:55:06.073845 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.073821 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-kube-api-access-5cq88" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "kube-api-access-5cq88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:06.074407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.074197 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-istio-token" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:06.074671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.074629 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-envoy" pod "7f585440-9878-4540-b806-6704375dc496" (UID: "7f585440-9878-4540-b806-6704375dc496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:06.174163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174128 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cq88\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-kube-api-access-5cq88\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174160 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174170 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f585440-9878-4540-b806-6704375dc496-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174178 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174186 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174194 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f585440-9878-4540-b806-6704375dc496-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174202 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174211 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f585440-9878-4540-b806-6704375dc496-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.174396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.174219 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f585440-9878-4540-b806-6704375dc496-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:06.243652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.243603 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:06.263164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.263135 6727 generic.go:356] "Generic (PLEG): container finished" podID="7f585440-9878-4540-b806-6704375dc496" containerID="b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc" exitCode=0 Oct 31 12:55:06.263598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.263192 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" Oct 31 12:55:06.263598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.263230 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" event={"ID":"7f585440-9878-4540-b806-6704375dc496","Type":"ContainerDied","Data":"b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc"} Oct 31 12:55:06.263598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.263310 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq" event={"ID":"7f585440-9878-4540-b806-6704375dc496","Type":"ContainerDied","Data":"2b3b8bf9dbda217ced0d760802b6cb5d1bd615f41638af1614adb4065c8800fb"} Oct 31 12:55:06.263598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.263350 6727 scope.go:122] "RemoveContainer" containerID="b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc" Oct 31 12:55:06.263598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.263484 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs" Oct 31 12:55:06.278856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.278835 6727 scope.go:122] "RemoveContainer" containerID="58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120" Oct 31 12:55:06.289819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.289799 6727 scope.go:122] "RemoveContainer" containerID="b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc" Oct 31 12:55:06.290129 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:06.290096 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc\": container with ID starting with b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc not found: ID does not exist" containerID="b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc" Oct 31 12:55:06.290231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.290132 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc"} err="failed to get container status \"b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc\": rpc error: code = NotFound desc = could not find container \"b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc\": container with ID starting with b8e688a209c4744a1ca639163f719a359dc7fb0d3023626bd81c6f74ceaecfbc not found: ID does not exist" Oct 31 12:55:06.290231 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.290151 6727 scope.go:122] "RemoveContainer" containerID="58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120" Oct 31 12:55:06.290426 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:06.290406 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120\": container with ID starting with 58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120 not found: ID does not exist" containerID="58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120" Oct 31 12:55:06.290488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.290434 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120"} err="failed to get container status \"58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120\": rpc error: code = NotFound desc = could not find container \"58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120\": container with ID starting with 58006d4ddf937f9a8a88ab20dd5428b19dc43abf07c1dffad912f518796cb120 not found: ID does not exist" Oct 31 12:55:06.300176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.300126 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq"] Oct 31 12:55:06.303317 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.303295 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d4dd68-openshift-default-779f75cf4f-b6mjq"] Oct 31 12:55:06.314240 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.314219 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs"] Oct 31 12:55:06.317642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.317623 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-4txgs"] Oct 31 12:55:06.391879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:06.391857 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:07.085952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:07.085921 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:07.302221 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:07.302191 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f585440-9878-4540-b806-6704375dc496" path="/var/lib/kubelet/pods/7f585440-9878-4540-b806-6704375dc496/volumes" Oct 31 12:55:07.302832 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:07.302812 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" path="/var/lib/kubelet/pods/89263210-c66c-4aea-8a82-0849d93f0a7a/volumes" Oct 31 12:55:07.303361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:07.303347 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" path="/var/lib/kubelet/pods/aafa41b6-0956-45e8-8b12-481bbbcaa965/volumes" Oct 31 12:55:07.816744 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:07.816713 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:08.580723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:08.580684 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:08.737876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:08.737846 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:09.262568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:09.262523 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:10.027185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.027147 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:10.242382 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.242343 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp"] Oct 31 12:55:10.243414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243393 6727 cpu_manager.go:414] "containerMap: removing container" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" Oct 31 12:55:10.243414 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243413 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243446 6727 cpu_manager.go:414] "containerMap: removing container" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243454 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243496 6727 cpu_manager.go:414] "containerMap: removing container" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243503 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243738 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="89263210-c66c-4aea-8a82-0849d93f0a7a" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243757 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243776 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" Oct 31 12:55:10.243893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.243791 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafa41b6-0956-45e8-8b12-481bbbcaa965" containerName="istio-proxy" Oct 31 12:55:10.249858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.249838 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.253883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.253861 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-71bc05\"/\"openshift-service-ca.crt\"" Oct 31 12:55:10.254591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.254569 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-71bc05\"/\"kube-root-ca.crt\"" Oct 31 12:55:10.255813 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.255794 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp"] Oct 31 12:55:10.258712 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.258653 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-71bc05\"/\"default-dockercfg-vb6v2\"" Oct 31 12:55:10.258786 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.258739 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-71bc05\"/\"test-model-71bc05-kserve-self-signed-certs\"" Oct 31 12:55:10.316496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.316390 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:10.330024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.329997 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-model-cache\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.330186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.330045 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-dshm\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.330186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.330172 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-tmp-dir\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.330351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.330334 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0076d52-4c70-4355-b8f1-7c90781701b6-tls-certs\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.330547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.330527 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5t9v\" (UniqueName: \"kubernetes.io/projected/b0076d52-4c70-4355-b8f1-7c90781701b6-kube-api-access-d5t9v\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.330665 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.330648 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-home\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.433483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.432433 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5t9v\" (UniqueName: \"kubernetes.io/projected/b0076d52-4c70-4355-b8f1-7c90781701b6-kube-api-access-d5t9v\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.433483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.432507 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-home\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.433483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.432559 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-model-cache\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.433483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.432606 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-dshm\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.433483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.432677 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-tmp-dir\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.433483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.432823 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0076d52-4c70-4355-b8f1-7c90781701b6-tls-certs\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.443364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.437951 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-model-cache\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.443364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.438696 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-home\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.443364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.440187 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-tmp-dir\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.444666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.444639 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0076d52-4c70-4355-b8f1-7c90781701b6-tls-certs\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.455225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.455196 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-dshm\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.472595 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.472562 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5t9v\" (UniqueName: \"kubernetes.io/projected/b0076d52-4c70-4355-b8f1-7c90781701b6-kube-api-access-d5t9v\") pod \"test-model-71bc05-kserve-f5475b959-64jzp\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.581687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.581657 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:10.583085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.583063 6727 scope.go:122] "RemoveContainer" containerID="fad0c3791b26a357167106109b39ce240a3db6f2da4fa72e0058d7dd0aa948f6" Oct 31 12:55:10.596365 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.596341 6727 scope.go:122] "RemoveContainer" containerID="1bbacec97c73367ecde0a2a7182859f46480d1c5bdd79b37113318e2309f66b6" Oct 31 12:55:10.886438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:10.886416 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp"] Oct 31 12:55:10.887999 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:55:10.887966 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0076d52_4c70_4355_b8f1_7c90781701b6.slice/crio-17bc8bf75999b322c894487fcddb64f42d71382c6d39065c0c48524bdfba70e5 WatchSource:0}: Error finding container 17bc8bf75999b322c894487fcddb64f42d71382c6d39065c0c48524bdfba70e5: Status 404 returned error can't find the container with id 17bc8bf75999b322c894487fcddb64f42d71382c6d39065c0c48524bdfba70e5 Oct 31 12:55:11.005669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.005634 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:11.312479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.312385 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" event={"ID":"b0076d52-4c70-4355-b8f1-7c90781701b6","Type":"ContainerStarted","Data":"0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060"} Oct 31 12:55:11.312479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.312427 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" event={"ID":"b0076d52-4c70-4355-b8f1-7c90781701b6","Type":"ContainerStarted","Data":"17bc8bf75999b322c894487fcddb64f42d71382c6d39065c0c48524bdfba70e5"} Oct 31 12:55:11.333809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.333766 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" podStartSLOduration=1.333752608 podStartE2EDuration="1.333752608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:11.331126611 +0000 UTC m=+1622.146141001" watchObservedRunningTime="2026-10-31 12:55:11.333752608 +0000 UTC m=+1622.148766995" Oct 31 12:55:11.538570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.538532 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:11.582314 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.582282 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:11.689699 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.689668 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:11.829927 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:11.829894 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:12.329301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:12.329274 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:12.341389 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:12.341365 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:12.772457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:12.772375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:13.338677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:13.338643 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:55:13.659800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:13.659719 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:14.473534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:14.473498 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:15.383677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:15.383625 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:16.121926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:16.121898 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:16.787718 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:16.787684 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:17.686144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:17.686103 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:18.383568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:18.383535 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:19.134944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:19.134910 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:19.845579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:19.845495 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:20.123142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:20.123062 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:20.581978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:20.581951 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:20.596115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:20.596076 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:21.069436 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.069403 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" containerID="cri-o://6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf" gracePeriod=45 Oct 31 12:55:21.318264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.318218 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:21.356607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.356583 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:55:21.400027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.399992 6727 generic.go:356] "Generic (PLEG): container finished" podID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerID="6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf" exitCode=0 Oct 31 12:55:21.407769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.401027 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" event={"ID":"44cd270f-5484-4e52-83bd-125f6bfa6f8d","Type":"ContainerDied","Data":"6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf"} Oct 31 12:55:21.407769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.404875 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" event={"ID":"44cd270f-5484-4e52-83bd-125f6bfa6f8d","Type":"ContainerDied","Data":"9ba5cecbb50a02df163dbf118c915bf007eeafd13997554752b8e33d2e0c44b7"} Oct 31 12:55:21.407769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.405073 6727 scope.go:122] "RemoveContainer" containerID="6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf" Oct 31 12:55:21.407769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.404207 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8" Oct 31 12:55:21.534736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.533714 6727 scope.go:122] "RemoveContainer" containerID="6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf" Oct 31 12:55:21.537681 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:21.537615 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf\": container with ID starting with 6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf not found: ID does not exist" containerID="6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf" Oct 31 12:55:21.537831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.537676 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf"} err="failed to get container status \"6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf\": rpc error: code = NotFound desc = could not find container \"6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf\": container with ID starting with 6d7ce928dfaafb8ef28fe0b295f1c534343984cf17466c0c478fafb089530fcf not found: ID does not exist" Oct 31 12:55:21.539475 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.539442 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:55:21.603333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603246 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tls-certs\") pod \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " Oct 31 12:55:21.603333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603303 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-dshm\") pod \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " Oct 31 12:55:21.603790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603387 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-model-cache\") pod \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " Oct 31 12:55:21.603790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603623 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-home\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-home\") pod \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " Oct 31 12:55:21.603790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603723 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-model-cache" pod "44cd270f-5484-4e52-83bd-125f6bfa6f8d" (UID: "44cd270f-5484-4e52-83bd-125f6bfa6f8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:21.603790 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603757 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44cd270f-5484-4e52-83bd-125f6bfa6f8d-kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/44cd270f-5484-4e52-83bd-125f6bfa6f8d-kube-api-access-pqd6z\") pod \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " Oct 31 12:55:21.603983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603819 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tmp-dir\") pod \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\" (UID: \"44cd270f-5484-4e52-83bd-125f6bfa6f8d\") " Oct 31 12:55:21.603983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.603891 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-home" pod "44cd270f-5484-4e52-83bd-125f6bfa6f8d" (UID: "44cd270f-5484-4e52-83bd-125f6bfa6f8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:21.604187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.604157 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tmp-dir" pod "44cd270f-5484-4e52-83bd-125f6bfa6f8d" (UID: "44cd270f-5484-4e52-83bd-125f6bfa6f8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:21.604433 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.604414 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:21.604650 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.604435 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:21.604650 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.604446 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:21.606058 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.606026 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tls-certs" pod "44cd270f-5484-4e52-83bd-125f6bfa6f8d" (UID: "44cd270f-5484-4e52-83bd-125f6bfa6f8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:21.606342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.606309 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cd270f-5484-4e52-83bd-125f6bfa6f8d-kube-api-access-pqd6z" pod "44cd270f-5484-4e52-83bd-125f6bfa6f8d" (UID: "44cd270f-5484-4e52-83bd-125f6bfa6f8d"). InnerVolumeSpecName "kube-api-access-pqd6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:21.606427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.606410 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-dshm" pod "44cd270f-5484-4e52-83bd-125f6bfa6f8d" (UID: "44cd270f-5484-4e52-83bd-125f6bfa6f8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:21.723538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.722665 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/44cd270f-5484-4e52-83bd-125f6bfa6f8d-kube-api-access-pqd6z\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:21.723538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.723140 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cd270f-5484-4e52-83bd-125f6bfa6f8d-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:21.723538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.723162 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cd270f-5484-4e52-83bd-125f6bfa6f8d-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:21.765667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.765636 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8"] Oct 31 12:55:21.771456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:21.771430 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-2-w2-e6c79d-kserve-86887dfd7b-sghs8"] Oct 31 12:55:22.294852 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:22.294820 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:23.174130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:23.169122 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:23.301848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:23.301813 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" path="/var/lib/kubelet/pods/44cd270f-5484-4e52-83bd-125f6bfa6f8d/volumes" Oct 31 12:55:24.138185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:24.137322 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:24.943634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:24.943588 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:25.746311 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:25.746280 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:26.725944 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:26.725905 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:27.560118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:27.560087 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:28.341134 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:28.341096 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:29.157978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:29.157940 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:29.608363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:29.608320 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:29.770834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:29.770793 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:30.147195 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:30.147144 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:30.342439 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:30.342407 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:55:30.938642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:30.938611 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:31.693277 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:31.693248 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:31.873485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:31.873396 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:32.718651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:32.718618 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:33.481843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:33.481812 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:34.003982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.003949 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9"] Oct 31 12:55:34.004752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.004715 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" podUID="50f9d07b-5a33-48d1-aa2a-db08ed4e5882" containerName="maas-api" containerID="cri-o://c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4" gracePeriod=30 Oct 31 12:55:34.080293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.080263 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:34.251730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.248766 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:34.413549 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.412699 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:34.447265 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.447239 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:55:34.504978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.504944 6727 generic.go:356] "Generic (PLEG): container finished" podID="50f9d07b-5a33-48d1-aa2a-db08ed4e5882" containerID="c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4" exitCode=0 Oct 31 12:55:34.505206 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.505171 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" event={"ID":"50f9d07b-5a33-48d1-aa2a-db08ed4e5882","Type":"ContainerDied","Data":"c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4"} Oct 31 12:55:34.505280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.505214 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" event={"ID":"50f9d07b-5a33-48d1-aa2a-db08ed4e5882","Type":"ContainerDied","Data":"d0977e6b76574b0c0c341c71c6d7fc95524bf487b021b3097b7ebf712404d2de"} Oct 31 12:55:34.505280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.505238 6727 scope.go:122] "RemoveContainer" containerID="c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4" Oct 31 12:55:34.505390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.505371 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9" Oct 31 12:55:34.521547 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.521523 6727 scope.go:122] "RemoveContainer" containerID="c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4" Oct 31 12:55:34.521939 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:34.521907 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4\": container with ID starting with c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4 not found: ID does not exist" containerID="c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4" Oct 31 12:55:34.522027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.521953 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4"} err="failed to get container status \"c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4\": rpc error: code = NotFound desc = could not find container \"c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4\": container with ID starting with c6e258e792b6b8ed2618d812a0a31e01e89b143e832319947f20146a5eeeabe4 not found: ID does not exist" Oct 31 12:55:34.529126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.529104 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-kube-api-access-gr74c\" (UniqueName: \"kubernetes.io/projected/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-kube-api-access-gr74c\") pod \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " Oct 31 12:55:34.529232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.529184 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-metrics-certs\") pod \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " Oct 31 12:55:34.529520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.529501 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-openshift-service-ca\") pod \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " Oct 31 12:55:34.529612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.529546 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls\") pod \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\" (UID: \"50f9d07b-5a33-48d1-aa2a-db08ed4e5882\") " Oct 31 12:55:34.530249 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.530225 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-openshift-service-ca" pod "50f9d07b-5a33-48d1-aa2a-db08ed4e5882" (UID: "50f9d07b-5a33-48d1-aa2a-db08ed4e5882"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:34.531848 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.531826 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-kube-api-access-gr74c" pod "50f9d07b-5a33-48d1-aa2a-db08ed4e5882" (UID: "50f9d07b-5a33-48d1-aa2a-db08ed4e5882"). InnerVolumeSpecName "kube-api-access-gr74c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:34.541630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.541604 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls" pod "50f9d07b-5a33-48d1-aa2a-db08ed4e5882" (UID: "50f9d07b-5a33-48d1-aa2a-db08ed4e5882"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:34.541784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.541624 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-metrics-certs" pod "50f9d07b-5a33-48d1-aa2a-db08ed4e5882" (UID: "50f9d07b-5a33-48d1-aa2a-db08ed4e5882"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:34.631552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.631522 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:34.631552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.631546 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:34.631552 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.631557 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr74c\" (UniqueName: \"kubernetes.io/projected/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-kube-api-access-gr74c\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:34.631772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.631566 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f9d07b-5a33-48d1-aa2a-db08ed4e5882-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:34.835731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.835702 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9"] Oct 31 12:55:34.847097 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.847066 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e6c79d-5fdfd5d879-g2pw9"] Oct 31 12:55:34.856674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.856648 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:34.998477 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:34.998447 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:35.302391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:35.302361 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f9d07b-5a33-48d1-aa2a-db08ed4e5882" path="/var/lib/kubelet/pods/50f9d07b-5a33-48d1-aa2a-db08ed4e5882/volumes" Oct 31 12:55:35.680700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:35.680672 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:36.411887 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:36.411859 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:37.151498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.150251 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb"] Oct 31 12:55:37.151753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.151708 6727 cpu_manager.go:414] "containerMap: removing container" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" Oct 31 12:55:37.151753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.151729 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f585440-9878-4540-b806-6704375dc496" containerName="istio-proxy" Oct 31 12:55:37.151876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.151776 6727 cpu_manager.go:414] "containerMap: removing container" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" Oct 31 12:55:37.151876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.151785 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" Oct 31 12:55:37.151876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.151799 6727 cpu_manager.go:414] "containerMap: removing container" podUID="50f9d07b-5a33-48d1-aa2a-db08ed4e5882" containerName="maas-api" Oct 31 12:55:37.151876 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.151807 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f9d07b-5a33-48d1-aa2a-db08ed4e5882" containerName="maas-api" Oct 31 12:55:37.152168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.152016 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cd270f-5484-4e52-83bd-125f6bfa6f8d" containerName="main" Oct 31 12:55:37.152168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.152030 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f9d07b-5a33-48d1-aa2a-db08ed4e5882" containerName="maas-api" Oct 31 12:55:37.159491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.156613 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:37.159491 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.157518 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.175490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.175447 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb"] Oct 31 12:55:37.197517 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.194799 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d086f3-openshift-default-dockercfg-hcwf6\"" Oct 31 12:55:37.268674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.268642 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.268840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.268684 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.268840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.268767 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.268840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.268830 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.268983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.268969 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqbz\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-kube-api-access-pkqbz\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.269018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.268991 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.269100 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.269088 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.269147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.269116 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.269147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.269143 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.370545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370508 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.370716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370563 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.370716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370593 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.370716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370646 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.370716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370677 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.370716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370712 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370778 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370927 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqbz\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-kube-api-access-pkqbz\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.370959 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371219 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.371109 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.371351 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.371377 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.371520 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.371571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.371540 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.372946 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.372920 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.373049 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.373022 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.382347 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.382326 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.382555 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.382533 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqbz\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-kube-api-access-pkqbz\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.484908 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.484839 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:37.772447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.772417 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb"] Oct 31 12:55:37.773327 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:55:37.773299 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8e0968_6f24_450b_8f8f_e7b00f1fb768.slice/crio-23ee615d6f45902f9b95eda60c8c324a3cd224944991177ee5ef7acf8964c04b WatchSource:0}: Error finding container 23ee615d6f45902f9b95eda60c8c324a3cd224944991177ee5ef7acf8964c04b: Status 404 returned error can't find the container with id 23ee615d6f45902f9b95eda60c8c324a3cd224944991177ee5ef7acf8964c04b Oct 31 12:55:37.776437 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.776398 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:55:37.776528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.776498 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:55:37.776573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:37.776543 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:55:38.031021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:38.030938 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:38.576921 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:38.576887 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" event={"ID":"7c8e0968-6f24-450b-8f8f-e7b00f1fb768","Type":"ContainerStarted","Data":"64f40430fcf60519d4442a051d8b546dfc5611a555366753e4d7a1f4d58ce28d"} Oct 31 12:55:38.581612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:38.581575 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" event={"ID":"7c8e0968-6f24-450b-8f8f-e7b00f1fb768","Type":"ContainerStarted","Data":"23ee615d6f45902f9b95eda60c8c324a3cd224944991177ee5ef7acf8964c04b"} Oct 31 12:55:38.720391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:38.720329 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podStartSLOduration=1.72031323 podStartE2EDuration="1.72031323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:38.66099007 +0000 UTC m=+1649.476004461" watchObservedRunningTime="2026-10-31 12:55:38.72031323 +0000 UTC m=+1649.535327619" Oct 31 12:55:39.131431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:39.131404 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:39.488370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:39.488290 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:39.488706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:39.488682 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 12:55:39.488870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:39.488848 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 12:55:40.006740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:40.006706 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:40.486193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:40.486159 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 12:55:40.486482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:40.486434 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 12:55:40.904505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:40.904299 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:41.487485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:41.486805 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 12:55:41.487485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:41.486860 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 12:55:41.809855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:41.809764 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:42.085093 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.085065 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:42.287977 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.287911 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:42.414930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.414817 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6"] Oct 31 12:55:42.415113 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.415048 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" containerID="cri-o://df17183ff35e66b4ff7c6c5a122cb02d832966a4c9c12e2ebcd0add1a66d6704" gracePeriod=30 Oct 31 12:55:42.421521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.419974 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw"] Oct 31 12:55:42.421521 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.420131 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" containerID="cri-o://e6c9312b1a42751430aa0f0fe2311fd978d9c95ed8399c622aee57834825c42b" gracePeriod=30 Oct 31 12:55:42.425189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.425163 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv"] Oct 31 12:55:42.445574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.445542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:42.486444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.485963 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 12:55:42.486444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.486044 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 12:55:42.503775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.503744 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:55:42.504120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.503806 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:55:42.643524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.629934 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:42.794934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.794850 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:42.938806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:42.938767 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:43.489164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:43.489132 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 12:55:43.489444 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:43.489419 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 12:55:43.646685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:43.646633 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:44.240729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:44.238226 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:44.431238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:44.430609 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:44.491071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:44.490993 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:44.491550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:44.491525 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:44.492307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:44.492286 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:55:44.596726 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:44.596693 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:44.792430 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:44.792352 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:45.444002 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:45.443964 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:45.725817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:45.725742 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:46.184657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:46.184624 6727 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:55:46.185077 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:46.184672 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:55:46.207691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:46.207623 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:46.963739 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:46.963705 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:47.427383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.427343 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" podUID="15eda72c-a4ed-408d-97d0-006037e7bd5f" containerName="payload-pre-processing" containerID="cri-o://2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429" gracePeriod=25 Oct 31 12:55:47.727455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.727420 6727 generic.go:356] "Generic (PLEG): container finished" podID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerID="e6c9312b1a42751430aa0f0fe2311fd978d9c95ed8399c622aee57834825c42b" exitCode=0 Oct 31 12:55:47.728453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.727593 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" event={"ID":"a97f8939-24d3-4f00-afd1-1238f8a59e98","Type":"ContainerDied","Data":"e6c9312b1a42751430aa0f0fe2311fd978d9c95ed8399c622aee57834825c42b"} Oct 31 12:55:47.730560 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.730440 6727 generic.go:356] "Generic (PLEG): container finished" podID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerID="df17183ff35e66b4ff7c6c5a122cb02d832966a4c9c12e2ebcd0add1a66d6704" exitCode=0 Oct 31 12:55:47.730729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.730604 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" event={"ID":"8199b71d-2f24-4d71-99c2-0282fdc31fca","Type":"ContainerDied","Data":"df17183ff35e66b4ff7c6c5a122cb02d832966a4c9c12e2ebcd0add1a66d6704"} Oct 31 12:55:47.730729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.730642 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" event={"ID":"8199b71d-2f24-4d71-99c2-0282fdc31fca","Type":"ContainerDied","Data":"aa63d2e7b539157c5c9dc3054daa9359d84bebd803680c6d03867a929237e747"} Oct 31 12:55:47.730729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.730659 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa63d2e7b539157c5c9dc3054daa9359d84bebd803680c6d03867a929237e747" Oct 31 12:55:47.730729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.730672 6727 scope.go:122] "RemoveContainer" containerID="00b190882d07c3100ac0711518918c9bd8a9d0e685b67276fb6b85a9314a16d7" Oct 31 12:55:47.747511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.747487 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:55:47.777814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.776707 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.780878 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-data\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781020 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-kube-api-access-cmnhc\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-kube-api-access-cmnhc\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781050 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-podinfo\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781067 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8199b71d-2f24-4d71-99c2-0282fdc31fca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8199b71d-2f24-4d71-99c2-0282fdc31fca-istiod-ca-cert\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781087 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-credential-socket\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781121 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-certs\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781178 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-socket\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781273 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-envoy\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.781583 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.781416 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-token\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-token\") pod \"8199b71d-2f24-4d71-99c2-0282fdc31fca\" (UID: \"8199b71d-2f24-4d71-99c2-0282fdc31fca\") " Oct 31 12:55:47.783395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.783205 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-socket" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.783879 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.783590 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-credential-socket" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.784054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.784024 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-certs" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.784498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.784167 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-data" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.785386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.785360 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8199b71d-2f24-4d71-99c2-0282fdc31fca-istiod-ca-cert" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:47.785598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.785579 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.785684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.785603 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8199b71d-2f24-4d71-99c2-0282fdc31fca-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.785684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.785617 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.785684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.785633 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.785684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.785646 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.787796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.787768 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-envoy" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.788312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.788289 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-kube-api-access-cmnhc" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "kube-api-access-cmnhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:47.789307 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.789276 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-token" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:47.790391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.790357 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-podinfo" pod "8199b71d-2f24-4d71-99c2-0282fdc31fca" (UID: "8199b71d-2f24-4d71-99c2-0282fdc31fca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:55:47.886528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.886263 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-credential-socket\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.886528 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.886329 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-kube-api-access-sn546\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-kube-api-access-sn546\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.886947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.886918 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-socket\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.886947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.886939 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-credential-socket" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.887132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887000 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-data\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.887132 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887072 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-token\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-token\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.887250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887154 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97f8939-24d3-4f00-afd1-1238f8a59e98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97f8939-24d3-4f00-afd1-1238f8a59e98-istiod-ca-cert\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.887250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887227 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-envoy\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.887367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887249 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-socket" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.887367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887310 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-podinfo\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.887367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887316 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-data" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.887367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887340 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-certs\") pod \"a97f8939-24d3-4f00-afd1-1238f8a59e98\" (UID: \"a97f8939-24d3-4f00-afd1-1238f8a59e98\") " Oct 31 12:55:47.887874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.887842 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97f8939-24d3-4f00-afd1-1238f8a59e98-istiod-ca-cert" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:47.888235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.888173 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-certs" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889552 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889584 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889602 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889616 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmnhc\" (UniqueName: \"kubernetes.io/projected/8199b71d-2f24-4d71-99c2-0282fdc31fca-kube-api-access-cmnhc\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889632 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889645 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8199b71d-2f24-4d71-99c2-0282fdc31fca-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889657 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889670 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.889722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.889683 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97f8939-24d3-4f00-afd1-1238f8a59e98-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.890860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.890783 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-kube-api-access-sn546" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "kube-api-access-sn546". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:47.890860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.890786 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-podinfo" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:55:47.891730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.891710 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-envoy" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.891959 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.891941 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-token" pod "a97f8939-24d3-4f00-afd1-1238f8a59e98" (UID: "a97f8939-24d3-4f00-afd1-1238f8a59e98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:47.925312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.925252 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:47.990912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.990831 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.990912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.990861 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn546\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-kube-api-access-sn546\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.990912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.990872 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.990912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:47.990881 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97f8939-24d3-4f00-afd1-1238f8a59e98-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:48.627863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.627822 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" podUID="15eda72c-a4ed-408d-97d0-006037e7bd5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:55:48.683509 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.667375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:48.738335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.738241 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" Oct 31 12:55:48.738335 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.738275 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw" event={"ID":"a97f8939-24d3-4f00-afd1-1238f8a59e98","Type":"ContainerDied","Data":"d5925fdb7e76c79b28a20aaf6ea21cfc687b45e26d8c934b53caf859c954b70f"} Oct 31 12:55:48.738584 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.738342 6727 scope.go:122] "RemoveContainer" containerID="e6c9312b1a42751430aa0f0fe2311fd978d9c95ed8399c622aee57834825c42b" Oct 31 12:55:48.739706 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.739687 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6" Oct 31 12:55:48.785814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.785787 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw"] Oct 31 12:55:48.793857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.793827 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-fl7tw"] Oct 31 12:55:48.806741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.806716 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6"] Oct 31 12:55:48.812230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:48.812209 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-z89r6"] Oct 31 12:55:49.302745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:49.302715 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" path="/var/lib/kubelet/pods/8199b71d-2f24-4d71-99c2-0282fdc31fca/volumes" Oct 31 12:55:49.303188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:49.303174 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" path="/var/lib/kubelet/pods/a97f8939-24d3-4f00-afd1-1238f8a59e98/volumes" Oct 31 12:55:49.482488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:49.480597 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:50.189400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:50.189316 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:51.200118 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:51.200076 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:52.068107 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:52.068073 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:52.672868 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:52.672831 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns"] Oct 31 12:55:52.673337 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:52.673065 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" podUID="afd5282b-eb99-4871-bc7f-44344133ff6d" containerName="maas-api" containerID="cri-o://14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc" gracePeriod=30 Oct 31 12:55:52.757565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:52.757529 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:52.918201 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:52.918173 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:53.061545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:53.060612 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:53.208714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:53.208684 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:53.471428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:53.471385 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:53.832953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:53.832916 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" podUID="afd5282b-eb99-4871-bc7f-44344133ff6d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.123:8443/health\": dial tcp 10.134.0.123:8443: connect: connection refused" Oct 31 12:55:53.955641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:53.955612 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:55:54.047432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.047370 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:54.080431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.080408 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd5282b-eb99-4871-bc7f-44344133ff6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5282b-eb99-4871-bc7f-44344133ff6d-openshift-service-ca\") pod \"afd5282b-eb99-4871-bc7f-44344133ff6d\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " Oct 31 12:55:54.080579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.080524 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd5282b-eb99-4871-bc7f-44344133ff6d-kube-api-access-5k864\" (UniqueName: \"kubernetes.io/projected/afd5282b-eb99-4871-bc7f-44344133ff6d-kube-api-access-5k864\") pod \"afd5282b-eb99-4871-bc7f-44344133ff6d\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " Oct 31 12:55:54.080579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.080554 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-metrics-certs\") pod \"afd5282b-eb99-4871-bc7f-44344133ff6d\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " Oct 31 12:55:54.080729 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.080712 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls\") pod \"afd5282b-eb99-4871-bc7f-44344133ff6d\" (UID: \"afd5282b-eb99-4871-bc7f-44344133ff6d\") " Oct 31 12:55:54.080805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.080781 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd5282b-eb99-4871-bc7f-44344133ff6d-openshift-service-ca" pod "afd5282b-eb99-4871-bc7f-44344133ff6d" (UID: "afd5282b-eb99-4871-bc7f-44344133ff6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:54.081190 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.081167 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5282b-eb99-4871-bc7f-44344133ff6d-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:54.082687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.082659 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd5282b-eb99-4871-bc7f-44344133ff6d-kube-api-access-5k864" pod "afd5282b-eb99-4871-bc7f-44344133ff6d" (UID: "afd5282b-eb99-4871-bc7f-44344133ff6d"). InnerVolumeSpecName "kube-api-access-5k864". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:54.082800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.082695 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls" pod "afd5282b-eb99-4871-bc7f-44344133ff6d" (UID: "afd5282b-eb99-4871-bc7f-44344133ff6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:54.082800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.082708 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-metrics-certs" pod "afd5282b-eb99-4871-bc7f-44344133ff6d" (UID: "afd5282b-eb99-4871-bc7f-44344133ff6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:54.182362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.182328 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k864\" (UniqueName: \"kubernetes.io/projected/afd5282b-eb99-4871-bc7f-44344133ff6d-kube-api-access-5k864\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:54.182362 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.182365 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:54.182613 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.182376 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5282b-eb99-4871-bc7f-44344133ff6d-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:54.793115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.790477 6727 generic.go:356] "Generic (PLEG): container finished" podID="afd5282b-eb99-4871-bc7f-44344133ff6d" containerID="14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc" exitCode=0 Oct 31 12:55:54.793115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.790764 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" event={"ID":"afd5282b-eb99-4871-bc7f-44344133ff6d","Type":"ContainerDied","Data":"14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc"} Oct 31 12:55:54.793115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.790860 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" event={"ID":"afd5282b-eb99-4871-bc7f-44344133ff6d","Type":"ContainerDied","Data":"f9812c3fb5e137ba6d8419c164876d1227d3755082f895c4746d355362a1b69f"} Oct 31 12:55:54.793115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.790896 6727 scope.go:122] "RemoveContainer" containerID="14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc" Oct 31 12:55:54.793115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.791085 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns" Oct 31 12:55:54.803474 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.803439 6727 scope.go:122] "RemoveContainer" containerID="14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc" Oct 31 12:55:54.803765 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:55:54.803745 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc\": container with ID starting with 14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc not found: ID does not exist" containerID="14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc" Oct 31 12:55:54.803840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.803774 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc"} err="failed to get container status \"14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc\": rpc error: code = NotFound desc = could not find container \"14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc\": container with ID starting with 14e74ccad5e37ac9146f84375804bab44b132c5cc48c5ff1d3c6e74e238b17dc not found: ID does not exist" Oct 31 12:55:54.820018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.819995 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns"] Oct 31 12:55:54.830668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.830644 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-015701-67cc6556dd-n6cns"] Oct 31 12:55:54.855003 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:54.854926 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:55.302330 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:55.302298 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd5282b-eb99-4871-bc7f-44344133ff6d" path="/var/lib/kubelet/pods/afd5282b-eb99-4871-bc7f-44344133ff6d/volumes" Oct 31 12:55:55.644072 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:55.644041 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:56.477209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:56.477131 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:57.234373 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:57.234336 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:57.957747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:57.957712 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:58.619523 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:58.619488 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:58.629183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:58.629064 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" podUID="15eda72c-a4ed-408d-97d0-006037e7bd5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:55:59.344217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:59.344174 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:59.409440 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:55:59.409411 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:56:00.342597 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:56:00.342561 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:56:00.389533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:00.389507 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:01.279127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:01.279092 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:02.032762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:02.032734 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:02.189851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:02.189810 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:02.776019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:02.775987 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:03.130453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:03.130423 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:03.365182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:03.365108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:04.255204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:04.255171 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:05.030268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:05.030234 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:06.026140 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:06.026109 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:06.754137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:06.754098 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:07.467388 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.466288 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:07.721585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.721561 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:56:07.781456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.781424 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15eda72c-a4ed-408d-97d0-006037e7bd5f-kube-api-access-xtsnc\" (UniqueName: \"kubernetes.io/projected/15eda72c-a4ed-408d-97d0-006037e7bd5f-kube-api-access-xtsnc\") pod \"15eda72c-a4ed-408d-97d0-006037e7bd5f\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " Oct 31 12:56:07.781637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.781565 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15eda72c-a4ed-408d-97d0-006037e7bd5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15eda72c-a4ed-408d-97d0-006037e7bd5f-plugins-config-volume\") pod \"15eda72c-a4ed-408d-97d0-006037e7bd5f\" (UID: \"15eda72c-a4ed-408d-97d0-006037e7bd5f\") " Oct 31 12:56:07.782007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.781972 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15eda72c-a4ed-408d-97d0-006037e7bd5f-plugins-config-volume" pod "15eda72c-a4ed-408d-97d0-006037e7bd5f" (UID: "15eda72c-a4ed-408d-97d0-006037e7bd5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:07.782302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.782277 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15eda72c-a4ed-408d-97d0-006037e7bd5f-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:07.784300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.784269 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15eda72c-a4ed-408d-97d0-006037e7bd5f-kube-api-access-xtsnc" pod "15eda72c-a4ed-408d-97d0-006037e7bd5f" (UID: "15eda72c-a4ed-408d-97d0-006037e7bd5f"). InnerVolumeSpecName "kube-api-access-xtsnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:07.883539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.883506 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtsnc\" (UniqueName: \"kubernetes.io/projected/15eda72c-a4ed-408d-97d0-006037e7bd5f-kube-api-access-xtsnc\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:07.895503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.895452 6727 generic.go:356] "Generic (PLEG): container finished" podID="15eda72c-a4ed-408d-97d0-006037e7bd5f" containerID="2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429" exitCode=0 Oct 31 12:56:07.895503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.895503 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" event={"ID":"15eda72c-a4ed-408d-97d0-006037e7bd5f","Type":"ContainerDied","Data":"2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429"} Oct 31 12:56:07.895694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.895511 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" Oct 31 12:56:07.895694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.895538 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv" event={"ID":"15eda72c-a4ed-408d-97d0-006037e7bd5f","Type":"ContainerDied","Data":"4d6cc09e01abe3d9fff43542574271d4603e891435f32506c15d74a0287f21d9"} Oct 31 12:56:07.895694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.895559 6727 scope.go:122] "RemoveContainer" containerID="2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429" Oct 31 12:56:07.909604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.909572 6727 scope.go:122] "RemoveContainer" containerID="2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429" Oct 31 12:56:07.909907 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:56:07.909879 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429\": container with ID starting with 2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429 not found: ID does not exist" containerID="2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429" Oct 31 12:56:07.909973 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.909922 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429"} err="failed to get container status \"2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429\": rpc error: code = NotFound desc = could not find container \"2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429\": container with ID starting with 2d0997c0e285f893d621d30eef9f944edd71096784972c05183b026ba500c429 not found: ID does not exist" Oct 31 12:56:07.921498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.921447 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv"] Oct 31 12:56:07.925581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:07.925562 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-015701-6c46845dc9-bl2nv"] Oct 31 12:56:08.214455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:08.214418 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:08.664360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:08.664311 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:08.965142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:08.965040 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:09.155432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:09.155378 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:09.302263 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:09.302194 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15eda72c-a4ed-408d-97d0-006037e7bd5f" path="/var/lib/kubelet/pods/15eda72c-a4ed-408d-97d0-006037e7bd5f/volumes" Oct 31 12:56:09.638624 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:09.638593 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:09.789803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:09.789765 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:10.377114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:10.377065 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:10.814655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:10.814581 6727 scope.go:122] "RemoveContainer" containerID="df17183ff35e66b4ff7c6c5a122cb02d832966a4c9c12e2ebcd0add1a66d6704" Oct 31 12:56:11.036068 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:11.036038 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:11.749524 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:11.749493 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:12.433084 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:12.433047 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:13.131126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:13.131093 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:13.664333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:13.664287 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:13.934630 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:13.934554 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj"] Oct 31 12:56:13.947640 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:13.947612 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg"] Oct 31 12:56:13.976565 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:13.976511 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:14.196711 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:14.196636 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:14.836693 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:14.836659 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:16.029932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:16.029901 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:16.340090 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:16.340041 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:16.978230 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:16.978199 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:17.666043 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:17.665994 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:18.360456 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:18.360421 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:18.935855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:18.935819 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" podUID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" containerName="payload-processing" containerID="cri-o://2f680eaa7365b01b405e29d4471e61cab0b3baa1ad37dba1476a43e2604eb75b" gracePeriod=25 Oct 31 12:56:18.948657 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:18.948628 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" podUID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" containerName="payload-pre-processing" containerID="cri-o://5dc66d2ce2f8c61da45370ce3c7d2986d4eb54d4a3951fd899a7a9a4d5055fc2" gracePeriod=25 Oct 31 12:56:19.063781 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:19.063747 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:19.854204 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:19.854164 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:20.554027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:20.553983 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:21.453038 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:21.453003 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:22.285981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.285949 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp"] Oct 31 12:56:22.297380 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297350 6727 cpu_manager.go:414] "containerMap: removing container" podUID="afd5282b-eb99-4871-bc7f-44344133ff6d" containerName="maas-api" Oct 31 12:56:22.297380 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297378 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd5282b-eb99-4871-bc7f-44344133ff6d" containerName="maas-api" Oct 31 12:56:22.297641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297392 6727 cpu_manager.go:414] "containerMap: removing container" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" Oct 31 12:56:22.297641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297400 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" Oct 31 12:56:22.297641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297433 6727 cpu_manager.go:414] "containerMap: removing container" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" Oct 31 12:56:22.297641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297441 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" Oct 31 12:56:22.297641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297456 6727 cpu_manager.go:414] "containerMap: removing container" podUID="15eda72c-a4ed-408d-97d0-006037e7bd5f" containerName="payload-pre-processing" Oct 31 12:56:22.297641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297479 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15eda72c-a4ed-408d-97d0-006037e7bd5f" containerName="payload-pre-processing" Oct 31 12:56:22.297861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297747 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd5282b-eb99-4871-bc7f-44344133ff6d" containerName="maas-api" Oct 31 12:56:22.297861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297769 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" Oct 31 12:56:22.297861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297781 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" Oct 31 12:56:22.297861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297793 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97f8939-24d3-4f00-afd1-1238f8a59e98" containerName="istio-proxy" Oct 31 12:56:22.297861 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.297802 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="15eda72c-a4ed-408d-97d0-006037e7bd5f" containerName="payload-pre-processing" Oct 31 12:56:22.306200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.305908 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf"] Oct 31 12:56:22.306200 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.306103 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.307603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.307577 6727 cpu_manager.go:414] "containerMap: removing container" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" Oct 31 12:56:22.307603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.307601 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8199b71d-2f24-4d71-99c2-0282fdc31fca" containerName="istio-proxy" Oct 31 12:56:22.312714 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.312688 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp"] Oct 31 12:56:22.313559 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.313528 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.314400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.314382 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:22.324752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.324710 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf"] Oct 31 12:56:22.398408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.398366 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.398580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.398524 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmw5\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-kube-api-access-2xmw5\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.398658 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.398635 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62233c53-e636-4b2d-b52f-2b9959ee47fd-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.398705 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.398671 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.398932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.398902 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.399078 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.398950 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.399449 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399424 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.399762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399744 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.399865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399796 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.399936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399889 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.399936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399927 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqnv\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-kube-api-access-4fqnv\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.400021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399952 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.400021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399972 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.400021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.399991 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.400021 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.400006 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.400149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.400052 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.400149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.400068 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.400149 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.400084 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.501686 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.501666 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmw5\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-kube-api-access-2xmw5\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.501800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.501715 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62233c53-e636-4b2d-b52f-2b9959ee47fd-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.501800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.501737 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.501800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.501760 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.501800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.501778 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.501806 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.501948 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502042 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502012 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502069 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502140 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqnv\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-kube-api-access-4fqnv\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502158 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502268 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502300 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502314 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502349 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502369 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502403 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502376 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502438 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502494 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502522 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.502779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502572 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62233c53-e636-4b2d-b52f-2b9959ee47fd-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502653 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502690 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.502779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502710 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.503197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.502903 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.503197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.503018 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.503197 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.503078 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.503333 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.503227 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.504564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.504541 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.504844 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.504827 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.504984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.504963 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.505103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.505085 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.515012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.514986 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmw5\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-kube-api-access-2xmw5\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.515343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.515316 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.515808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.515786 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.517486 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.517452 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqnv\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-kube-api-access-4fqnv\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.652176 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.652157 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:22.661316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.661298 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:22.954833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.954803 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp"] Oct 31 12:56:22.955438 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:56:22.955374 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62233c53_e636_4b2d_b52f_2b9959ee47fd.slice/crio-ced9ed90a02e2e3214d025858af718399dd2a90ca78c77d2dc247561c1748e61 WatchSource:0}: Error finding container ced9ed90a02e2e3214d025858af718399dd2a90ca78c77d2dc247561c1748e61: Status 404 returned error can't find the container with id ced9ed90a02e2e3214d025858af718399dd2a90ca78c77d2dc247561c1748e61 Oct 31 12:56:22.958562 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.958537 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf"] Oct 31 12:56:22.959039 ip-10-0-142-45 kubenswrapper[6727]: W1031 12:56:22.959012 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45cda06_0d2f_45f4_917f_e854e4cb30ba.slice/crio-6bcd2fd63e0ad69fee4aa358fabe25edcb231007283b567a93b83c3f498b6652 WatchSource:0}: Error finding container 6bcd2fd63e0ad69fee4aa358fabe25edcb231007283b567a93b83c3f498b6652: Status 404 returned error can't find the container with id 6bcd2fd63e0ad69fee4aa358fabe25edcb231007283b567a93b83c3f498b6652 Oct 31 12:56:22.959284 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.959230 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:56:22.959391 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.959333 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:56:22.959476 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.959402 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:56:22.961570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.961539 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:56:22.961674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.961610 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:56:22.961674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.961655 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 12:56:22.999116 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:22.999089 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" event={"ID":"62233c53-e636-4b2d-b52f-2b9959ee47fd","Type":"ContainerStarted","Data":"ced9ed90a02e2e3214d025858af718399dd2a90ca78c77d2dc247561c1748e61"} Oct 31 12:56:23.000392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:23.000369 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" event={"ID":"f45cda06-0d2f-45f4-917f-e854e4cb30ba","Type":"ContainerStarted","Data":"6bcd2fd63e0ad69fee4aa358fabe25edcb231007283b567a93b83c3f498b6652"} Oct 31 12:56:23.167538 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:23.166378 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:24.008296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.008251 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" event={"ID":"f45cda06-0d2f-45f4-917f-e854e4cb30ba","Type":"ContainerStarted","Data":"f6c60dbf774bce16593610611bfbde684d7ca47d3ee47d9f1530f09d7ebde4d1"} Oct 31 12:56:24.010136 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.010104 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" event={"ID":"62233c53-e636-4b2d-b52f-2b9959ee47fd","Type":"ContainerStarted","Data":"abbc0dfc7fc9e08e8f574531ed5879116f9307ad40743a52cde8a437911f82bb"} Oct 31 12:56:24.012952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.012925 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:24.046563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.043427 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podStartSLOduration=2.043406447 podStartE2EDuration="2.043406447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:24.040937275 +0000 UTC m=+1694.855951666" watchObservedRunningTime="2026-10-31 12:56:24.043406447 +0000 UTC m=+1694.858420837" Oct 31 12:56:24.072482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.070425 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podStartSLOduration=2.070406627 podStartE2EDuration="2.070406627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:24.066952591 +0000 UTC m=+1694.881966986" watchObservedRunningTime="2026-10-31 12:56:24.070406627 +0000 UTC m=+1694.885421022" Oct 31 12:56:24.648960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.648929 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:24.654129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.654101 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:24.655610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.655535 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 12:56:24.655610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.655600 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 12:56:24.661640 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.661613 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:24.683505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.683164 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 12:56:24.683505 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.683226 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 12:56:24.880487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:24.877802 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:25.080051 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.080013 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:25.290963 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.290932 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:25.437522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.431320 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" podUID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:56:25.437522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.433199 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" podUID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:56:25.526483 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.525529 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:25.653156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.653113 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 12:56:25.653348 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.653176 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 12:56:25.661855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.661829 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 12:56:25.661999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:25.661865 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 12:56:26.428008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:26.427967 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:26.652568 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:26.652539 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 12:56:26.652731 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:26.652588 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 12:56:26.661616 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:26.661589 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 12:56:26.661703 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:26.661666 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 12:56:27.333634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:27.333600 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:27.652801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:27.652717 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 12:56:27.652801 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:27.652767 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 12:56:27.662511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:27.662478 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 12:56:27.662676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:27.662526 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 12:56:28.219488 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:28.219158 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:28.653108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:28.653067 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 12:56:28.653599 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:28.653130 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 12:56:28.662196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:28.662168 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 12:56:28.662316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:28.662215 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 12:56:29.215490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:29.211289 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:29.653081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:29.653049 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 12:56:29.653260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:29.653099 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 12:56:29.661697 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:29.661670 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 12:56:29.661797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:29.661722 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 12:56:29.946764 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:29.946672 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:30.343397 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:56:30.343361 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:56:30.660749 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:30.660669 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:30.665130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:30.665074 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:30.667252 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:30.667216 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:56:30.669641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:30.669621 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:30.669937 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:30.669916 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:30.671022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:30.670999 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:56:30.743139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:30.743108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:31.531484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:31.531376 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:32.214363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:32.214322 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:33.036453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:33.036419 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:33.224590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:33.224560 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:33.395245 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:33.395209 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:33.863056 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:33.863026 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:34.619506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:34.619438 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:35.385503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:35.385455 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" podUID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:56:35.385687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:35.385601 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" podUID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:56:35.435246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:35.435213 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:36.277057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:36.277028 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:37.163744 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:37.163682 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:37.835007 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:37.834932 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:38.616981 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:38.616938 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:38.705071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:38.705034 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq"] Oct 31 12:56:38.723084 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:38.723044 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv"] Oct 31 12:56:38.884549 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:38.884432 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:39.220595 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.220563 6727 generic.go:356] "Generic (PLEG): container finished" podID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" containerID="2f680eaa7365b01b405e29d4471e61cab0b3baa1ad37dba1476a43e2604eb75b" exitCode=0 Oct 31 12:56:39.220740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.220637 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" event={"ID":"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e","Type":"ContainerDied","Data":"2f680eaa7365b01b405e29d4471e61cab0b3baa1ad37dba1476a43e2604eb75b"} Oct 31 12:56:39.220740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.220701 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" event={"ID":"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e","Type":"ContainerDied","Data":"3d7c9ad52e22e764ffcf6dfc253fe3c012919755ea5677cc40a7d28921a556d0"} Oct 31 12:56:39.220740 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.220728 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d7c9ad52e22e764ffcf6dfc253fe3c012919755ea5677cc40a7d28921a556d0" Oct 31 12:56:39.222564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.222538 6727 generic.go:356] "Generic (PLEG): container finished" podID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" containerID="5dc66d2ce2f8c61da45370ce3c7d2986d4eb54d4a3951fd899a7a9a4d5055fc2" exitCode=0 Oct 31 12:56:39.222765 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.222748 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" event={"ID":"bdc0a033-4793-43ba-bb55-a960b69cf7dc","Type":"ContainerDied","Data":"5dc66d2ce2f8c61da45370ce3c7d2986d4eb54d4a3951fd899a7a9a4d5055fc2"} Oct 31 12:56:39.236755 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.236732 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:56:39.278708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.278680 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:56:39.310581 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.310551 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-kube-api-access-b22db\" (UniqueName: \"kubernetes.io/projected/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-kube-api-access-b22db\") pod \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " Oct 31 12:56:39.311066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.310820 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-plugins-config-volume\") pod \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\" (UID: \"9fb6e92d-9d0a-41fc-bd1f-8ace2848716e\") " Oct 31 12:56:39.311238 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.311195 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-plugins-config-volume" pod "9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" (UID: "9fb6e92d-9d0a-41fc-bd1f-8ace2848716e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:39.313691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.311995 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:39.315873 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.315847 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-kube-api-access-b22db" pod "9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" (UID: "9fb6e92d-9d0a-41fc-bd1f-8ace2848716e"). InnerVolumeSpecName "kube-api-access-b22db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:39.412762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.412706 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc0a033-4793-43ba-bb55-a960b69cf7dc-kube-api-access-sr6fm\" (UniqueName: \"kubernetes.io/projected/bdc0a033-4793-43ba-bb55-a960b69cf7dc-kube-api-access-sr6fm\") pod \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " Oct 31 12:56:39.412762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.412758 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc0a033-4793-43ba-bb55-a960b69cf7dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc0a033-4793-43ba-bb55-a960b69cf7dc-plugins-config-volume\") pod \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\" (UID: \"bdc0a033-4793-43ba-bb55-a960b69cf7dc\") " Oct 31 12:56:39.413182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.413157 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc0a033-4793-43ba-bb55-a960b69cf7dc-plugins-config-volume" pod "bdc0a033-4793-43ba-bb55-a960b69cf7dc" (UID: "bdc0a033-4793-43ba-bb55-a960b69cf7dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:39.413268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.413216 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b22db\" (UniqueName: \"kubernetes.io/projected/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e-kube-api-access-b22db\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:39.414688 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.414664 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc0a033-4793-43ba-bb55-a960b69cf7dc-kube-api-access-sr6fm" pod "bdc0a033-4793-43ba-bb55-a960b69cf7dc" (UID: "bdc0a033-4793-43ba-bb55-a960b69cf7dc"). InnerVolumeSpecName "kube-api-access-sr6fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:39.447772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.447739 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:39.514292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.514257 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr6fm\" (UniqueName: \"kubernetes.io/projected/bdc0a033-4793-43ba-bb55-a960b69cf7dc-kube-api-access-sr6fm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:39.514292 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:39.514285 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc0a033-4793-43ba-bb55-a960b69cf7dc-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:40.232516 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.232402 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj" Oct 31 12:56:40.233598 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.233573 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" Oct 31 12:56:40.237590 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.237545 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg" event={"ID":"bdc0a033-4793-43ba-bb55-a960b69cf7dc","Type":"ContainerDied","Data":"beab81599443f55d45d6ab86fd545322a9ccb0e8b58da91c8d1c46d81ef023c5"} Oct 31 12:56:40.237754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.237655 6727 scope.go:122] "RemoveContainer" containerID="5dc66d2ce2f8c61da45370ce3c7d2986d4eb54d4a3951fd899a7a9a4d5055fc2" Oct 31 12:56:40.271730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.271645 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:40.294199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.294170 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg"] Oct 31 12:56:40.297313 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.297288 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ad1af7-6cbf6f4f59-j2tmg"] Oct 31 12:56:40.305418 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.305398 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj"] Oct 31 12:56:40.309808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:40.309789 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ad1af7-764fbc6d9f-f5qsj"] Oct 31 12:56:41.007513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:41.007459 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:41.301819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:41.301743 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" path="/var/lib/kubelet/pods/9fb6e92d-9d0a-41fc-bd1f-8ace2848716e/volumes" Oct 31 12:56:41.302227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:41.302201 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" path="/var/lib/kubelet/pods/bdc0a033-4793-43ba-bb55-a960b69cf7dc/volumes" Oct 31 12:56:41.651044 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:41.651010 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:42.372129 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:42.372086 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:43.160181 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:43.160146 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:43.861061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:43.861029 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:44.153147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:44.153062 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:44.285092 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:44.285064 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:44.731203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:44.730409 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:45.427022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:45.416919 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:46.086145 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:46.086113 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:46.630977 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:46.630947 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs"] Oct 31 12:56:46.631428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:46.631203 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" containerID="cri-o://280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6" gracePeriod=30 Oct 31 12:56:46.727188 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:46.727142 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:47.413209 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:47.413174 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:48.032268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:48.032237 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:56:48.260727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:48.260685 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:49.298785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:49.298751 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:50.220105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:50.220008 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:51.245096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:51.245057 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:51.908970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:51.908914 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:56:52.043969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.043931 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095de43a-2809-4ebe-aaf1-208d47b55993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095de43a-2809-4ebe-aaf1-208d47b55993-istiod-ca-cert\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.043984 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-envoy\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044018 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-kube-api-access-w6l4l\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-kube-api-access-w6l4l\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044052 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-socket\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044150 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044091 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/095de43a-2809-4ebe-aaf1-208d47b55993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095de43a-2809-4ebe-aaf1-208d47b55993-istio-podinfo\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044200 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-data\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044245 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-credential-socket\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044317 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095de43a-2809-4ebe-aaf1-208d47b55993-istiod-ca-cert" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:52.044410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044344 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-certs\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044497 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-istio-token\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-istio-token\") pod \"095de43a-2809-4ebe-aaf1-208d47b55993\" (UID: \"095de43a-2809-4ebe-aaf1-208d47b55993\") " Oct 31 12:56:52.044641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044516 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-socket" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:52.044751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044688 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-credential-socket" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:52.044751 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044725 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-certs" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:52.044866 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.044843 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-data" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:52.045234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.045209 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095de43a-2809-4ebe-aaf1-208d47b55993-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.045338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.045237 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.045338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.045254 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.045338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.045266 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.045338 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.045280 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.046504 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.046442 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-envoy" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:52.047293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.046968 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/095de43a-2809-4ebe-aaf1-208d47b55993-istio-podinfo" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:56:52.047293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.047172 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-istio-token" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:52.047858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.047838 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-kube-api-access-w6l4l" pod "095de43a-2809-4ebe-aaf1-208d47b55993" (UID: "095de43a-2809-4ebe-aaf1-208d47b55993"). InnerVolumeSpecName "kube-api-access-w6l4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:52.146937 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.146905 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095de43a-2809-4ebe-aaf1-208d47b55993-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.146937 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.146932 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6l4l\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-kube-api-access-w6l4l\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.147072 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.146948 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095de43a-2809-4ebe-aaf1-208d47b55993-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.147072 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.146960 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095de43a-2809-4ebe-aaf1-208d47b55993-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:52.245641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.245527 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:52.361510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.361476 6727 generic.go:356] "Generic (PLEG): container finished" podID="095de43a-2809-4ebe-aaf1-208d47b55993" containerID="280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6" exitCode=0 Oct 31 12:56:52.361674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.361548 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" Oct 31 12:56:52.361674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.361551 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" event={"ID":"095de43a-2809-4ebe-aaf1-208d47b55993","Type":"ContainerDied","Data":"280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6"} Oct 31 12:56:52.361674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.361601 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs" event={"ID":"095de43a-2809-4ebe-aaf1-208d47b55993","Type":"ContainerDied","Data":"dcfd62cfda346b9c87adbeaf440981df4ee81620862f50b8e6868be491d791ac"} Oct 31 12:56:52.361674 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.361624 6727 scope.go:122] "RemoveContainer" containerID="280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6" Oct 31 12:56:52.372543 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.372527 6727 scope.go:122] "RemoveContainer" containerID="280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6" Oct 31 12:56:52.372783 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:56:52.372766 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6\": container with ID starting with 280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6 not found: ID does not exist" containerID="280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6" Oct 31 12:56:52.372833 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.372789 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6"} err="failed to get container status \"280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6\": rpc error: code = NotFound desc = could not find container \"280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6\": container with ID starting with 280bed77319a585165935a39de2ed08761731820f1ba4d9c4611870d94295ac6 not found: ID does not exist" Oct 31 12:56:52.388811 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.388790 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs"] Oct 31 12:56:52.397512 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:52.397493 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-897xs"] Oct 31 12:56:53.152224 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:53.152194 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:53.302964 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:53.302933 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" path="/var/lib/kubelet/pods/095de43a-2809-4ebe-aaf1-208d47b55993/volumes" Oct 31 12:56:53.727283 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:53.727243 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" containerID="cri-o://f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7" gracePeriod=45 Oct 31 12:56:53.744736 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:53.744697 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" containerID="cri-o://4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d" gracePeriod=45 Oct 31 12:56:53.819650 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:53.819595 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:53.903649 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:53.901363 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 31 12:56:54.061367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.061343 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:56:54.065277 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.065258 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:56:54.176920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.176891 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-dshm\") pod \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " Oct 31 12:56:54.177104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.176948 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tls-certs\") pod \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " Oct 31 12:56:54.177104 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177030 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543a8bb9-7cd3-4055-8109-7fb9d60590a8-kube-api-access-wqzcg\" (UniqueName: \"kubernetes.io/projected/543a8bb9-7cd3-4055-8109-7fb9d60590a8-kube-api-access-wqzcg\") pod \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " Oct 31 12:56:54.177228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177120 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tmp-dir\") pod \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " Oct 31 12:56:54.177228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177170 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-home\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-home\") pod \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " Oct 31 12:56:54.177228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177201 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tls-certs\" (UniqueName: \"kubernetes.io/secret/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tls-certs\") pod \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " Oct 31 12:56:54.177387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177231 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-home\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-home\") pod \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " Oct 31 12:56:54.177387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177303 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-dshm\") pod \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " Oct 31 12:56:54.177387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177368 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-model-cache\") pod \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " Oct 31 12:56:54.177618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177432 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tmp-dir\") pod \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " Oct 31 12:56:54.177618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177490 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-kube-api-access-lvrw4\" (UniqueName: \"kubernetes.io/projected/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-kube-api-access-lvrw4\") pod \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\" (UID: \"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16\") " Oct 31 12:56:54.177618 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177524 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tmp-dir" pod "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" (UID: "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.177752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177665 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-model-cache\") pod \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\" (UID: \"543a8bb9-7cd3-4055-8109-7fb9d60590a8\") " Oct 31 12:56:54.177793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.177761 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-model-cache" pod "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" (UID: "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.178052 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.178019 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-home" pod "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" (UID: "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.178248 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.178223 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tmp-dir" pod "543a8bb9-7cd3-4055-8109-7fb9d60590a8" (UID: "543a8bb9-7cd3-4055-8109-7fb9d60590a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.178646 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.178586 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-home" pod "543a8bb9-7cd3-4055-8109-7fb9d60590a8" (UID: "543a8bb9-7cd3-4055-8109-7fb9d60590a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.179025 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.178992 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-model-cache" pod "543a8bb9-7cd3-4055-8109-7fb9d60590a8" (UID: "543a8bb9-7cd3-4055-8109-7fb9d60590a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.180023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.179910 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.180023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.179936 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.180023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.179949 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.180023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.179963 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.180023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.179986 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.180023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.179997 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.180383 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.180357 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-dshm" pod "543a8bb9-7cd3-4055-8109-7fb9d60590a8" (UID: "543a8bb9-7cd3-4055-8109-7fb9d60590a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.181439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.181412 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-kube-api-access-lvrw4" pod "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" (UID: "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16"). InnerVolumeSpecName "kube-api-access-lvrw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:54.181989 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.181930 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-dshm" pod "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" (UID: "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:56:54.182158 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.182134 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tls-certs" pod "543a8bb9-7cd3-4055-8109-7fb9d60590a8" (UID: "543a8bb9-7cd3-4055-8109-7fb9d60590a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:54.183426 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.183402 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tls-certs" pod "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" (UID: "65c629cd-c1e0-4fbd-8b03-1eea66a6dc16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:54.184574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.184556 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543a8bb9-7cd3-4055-8109-7fb9d60590a8-kube-api-access-wqzcg" pod "543a8bb9-7cd3-4055-8109-7fb9d60590a8" (UID: "543a8bb9-7cd3-4055-8109-7fb9d60590a8"). InnerVolumeSpecName "kube-api-access-wqzcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:54.281830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.281746 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzcg\" (UniqueName: \"kubernetes.io/projected/543a8bb9-7cd3-4055-8109-7fb9d60590a8-kube-api-access-wqzcg\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.281830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.281777 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.281830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.281791 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8bb9-7cd3-4055-8109-7fb9d60590a8-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.281830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.281802 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrw4\" (UniqueName: \"kubernetes.io/projected/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-kube-api-access-lvrw4\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.281830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.281814 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.281830 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.281827 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8bb9-7cd3-4055-8109-7fb9d60590a8-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:54.376946 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.376914 6727 generic.go:356] "Generic (PLEG): container finished" podID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerID="4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d" exitCode=0 Oct 31 12:56:54.377323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.376956 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" Oct 31 12:56:54.377323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.376996 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" event={"ID":"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16","Type":"ContainerDied","Data":"4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d"} Oct 31 12:56:54.377323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.377036 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" event={"ID":"65c629cd-c1e0-4fbd-8b03-1eea66a6dc16","Type":"ContainerDied","Data":"8a5a40386b52f5edb437f140981a1dc6ef62539b2cad2188320db1c61399706a"} Oct 31 12:56:54.377323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.377058 6727 scope.go:122] "RemoveContainer" containerID="4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d" Oct 31 12:56:54.378677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.378652 6727 generic.go:356] "Generic (PLEG): container finished" podID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerID="f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7" exitCode=0 Oct 31 12:56:54.378780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.378698 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" Oct 31 12:56:54.378780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.378742 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" event={"ID":"543a8bb9-7cd3-4055-8109-7fb9d60590a8","Type":"ContainerDied","Data":"f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7"} Oct 31 12:56:54.378780 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.378771 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq" event={"ID":"543a8bb9-7cd3-4055-8109-7fb9d60590a8","Type":"ContainerDied","Data":"a97b9166fd29140f7b28666ab5c15fe1c0a196b3a0e32d7857c6425627b9af3b"} Oct 31 12:56:54.389193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.389171 6727 scope.go:122] "RemoveContainer" containerID="4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d" Oct 31 12:56:54.389665 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:56:54.389639 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d\": container with ID starting with 4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d not found: ID does not exist" containerID="4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d" Oct 31 12:56:54.389769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.389681 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d"} err="failed to get container status \"4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d\": rpc error: code = NotFound desc = could not find container \"4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d\": container with ID starting with 4951d142fac087396e7eaec806967f61f958bf5b1cfee91702af1c3384c72a9d not found: ID does not exist" Oct 31 12:56:54.389769 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.389708 6727 scope.go:122] "RemoveContainer" containerID="f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7" Oct 31 12:56:54.401578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.401561 6727 scope.go:122] "RemoveContainer" containerID="f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7" Oct 31 12:56:54.401876 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:56:54.401856 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7\": container with ID starting with f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7 not found: ID does not exist" containerID="f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7" Oct 31 12:56:54.401932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.401892 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7"} err="failed to get container status \"f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7\": rpc error: code = NotFound desc = could not find container \"f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7\": container with ID starting with f77371ca72c69f21bbef7a8030f0569a57e5e6b1d8d8c18f2a4af3d86c36aee7 not found: ID does not exist" Oct 31 12:56:54.407977 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.407955 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv"] Oct 31 12:56:54.413019 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.413000 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv"] Oct 31 12:56:54.423126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.423105 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq"] Oct 31 12:56:54.427522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.427502 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-w0-fb0343-kserve-68b6b8dfcc-x4fnq"] Oct 31 12:56:54.541838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:54.541749 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:55.055687 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:55.055651 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/simulator-w0-fb0343-kserve-85557484b7-w8gkv" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:55.301340 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:55.301311 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" path="/var/lib/kubelet/pods/543a8bb9-7cd3-4055-8109-7fb9d60590a8/volumes" Oct 31 12:56:55.301798 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:55.301782 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" path="/var/lib/kubelet/pods/65c629cd-c1e0-4fbd-8b03-1eea66a6dc16/volumes" Oct 31 12:56:55.314570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:55.314496 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:56.211170 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:56.211139 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:56.923279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:56.923241 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:57.649453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:57.649417 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:57.798203 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:57.798173 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:58.355793 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:58.352734 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:59.059009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:59.058981 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:59.726784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:56:59.726753 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:00.342055 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:57:00.342017 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:57:00.541857 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:00.540060 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:01.323637 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:01.323592 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:02.016758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:02.016730 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:02.684695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:02.684666 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:03.450126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:03.450085 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:04.143117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:04.143064 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:04.826803 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:04.826767 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:05.526926 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:05.526888 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:06.093522 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:06.093492 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:06.242457 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:06.242426 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:06.396219 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:06.396149 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:06.539226 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:06.539189 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:06.840713 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:06.840680 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:07.135659 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:07.135568 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:07.842976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:07.842932 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:08.548360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:08.548328 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:08.784232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:08.784195 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:09.196632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:09.196598 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:09.324380 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:09.324337 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:10.028508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:10.028431 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:10.750643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:10.750612 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:10.863377 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:10.863349 6727 scope.go:122] "RemoveContainer" containerID="2f680eaa7365b01b405e29d4471e61cab0b3baa1ad37dba1476a43e2604eb75b" Oct 31 12:57:11.518623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:11.518588 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:12.503056 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:12.503021 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:13.243865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:13.243830 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:14.063727 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:14.063692 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:14.980415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:14.980380 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:15.269054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.268977 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:15.442817 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.442778 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:15.742304 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.742238 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx"] Oct 31 12:57:15.742567 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.742541 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" containerID="cri-o://cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4" gracePeriod=30 Oct 31 12:57:15.782330 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.782302 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:15.934932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.934897 6727 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:57:15.935220 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.935191 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:57:15.962576 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:15.962264 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:16.717243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:16.717196 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:16.923788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:16.923760 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:17.100267 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:17.100238 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:17.335312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:17.335271 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:17.510564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:17.507098 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:17.725893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:17.725861 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:18.555575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:18.555538 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:18.915741 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:18.915710 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:19.477672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:19.477644 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:20.261965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:20.261930 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:21.024472 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.024409 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:57:21.084402 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.084372 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:21.102120 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.102063 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-envoy\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.102309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.102160 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-socket\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.102309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.102189 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istiod-ca-cert\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.102428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.102307 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-kube-api-access-9dtv2\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-kube-api-access-9dtv2\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.102901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.102835 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-token\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-token\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.103363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.103319 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-credential-socket\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.103540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.103366 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-podinfo\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.103540 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.103422 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-certs\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.103754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.103555 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-data\") pod \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\" (UID: \"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce\") " Oct 31 12:57:21.104698 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.104675 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-data" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:57:21.107108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.107082 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istiod-ca-cert" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:57:21.107108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.107340 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-socket" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:57:21.107108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.107934 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-certs" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:57:21.107108 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.108444 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-credential-socket" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:57:21.115660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.113498 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-token" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:57:21.115660 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.113644 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-envoy" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:57:21.120408 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.120353 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-podinfo" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:57:21.120655 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.120622 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-kube-api-access-9dtv2" pod "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" (UID: "7ad04ba7-e99f-4779-95fd-e0ebd5e889ce"). InnerVolumeSpecName "kube-api-access-9dtv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:57:21.205661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205626 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205656 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205667 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205675 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205684 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205692 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205703 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205715 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dtv2\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-kube-api-access-9dtv2\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.205930 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.205724 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:21.599442 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.599404 6727 generic.go:356] "Generic (PLEG): container finished" podID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerID="cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4" exitCode=0 Oct 31 12:57:21.599891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.599457 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" Oct 31 12:57:21.599891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.599520 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" event={"ID":"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce","Type":"ContainerDied","Data":"cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4"} Oct 31 12:57:21.599891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.599577 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx" event={"ID":"7ad04ba7-e99f-4779-95fd-e0ebd5e889ce","Type":"ContainerDied","Data":"cf32ace922ea517dc4a7f602825c6a261779882ab6f899f161feeee0fc9741dd"} Oct 31 12:57:21.599891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.599598 6727 scope.go:122] "RemoveContainer" containerID="cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4" Oct 31 12:57:21.617746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.617724 6727 scope.go:122] "RemoveContainer" containerID="cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4" Oct 31 12:57:21.618021 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:57:21.617997 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4\": container with ID starting with cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4 not found: ID does not exist" containerID="cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4" Oct 31 12:57:21.618162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.618038 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4"} err="failed to get container status \"cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4\": rpc error: code = NotFound desc = could not find container \"cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4\": container with ID starting with cd9720be3503e0d786f0d8b9d8e866d947093331c42697e848566c83cd42f9f4 not found: ID does not exist" Oct 31 12:57:21.625894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.625870 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx"] Oct 31 12:57:21.632756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.632736 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-8wzzx"] Oct 31 12:57:21.911260 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:21.911187 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:22.616899 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:22.616865 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:23.301695 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:23.301666 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" path="/var/lib/kubelet/pods/7ad04ba7-e99f-4779-95fd-e0ebd5e889ce/volumes" Oct 31 12:57:23.458349 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:23.458286 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:24.234268 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:24.234209 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:25.003269 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:25.003240 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:25.604821 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:25.604793 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:25.738615 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:25.738578 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:26.322493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:26.322441 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:27.024267 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:27.024230 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:27.659027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:27.658944 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:28.429470 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:28.429427 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:29.162949 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:29.161179 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:29.926652 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:29.926618 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:30.342509 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:57:30.342436 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:57:30.628883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:30.628762 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:31.277269 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:31.277237 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:31.919551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:31.919518 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:32.631764 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:32.631731 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:33.280907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:33.280875 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:33.913404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:33.913372 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:34.638251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:34.638215 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:35.268386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:35.268350 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:57:35.349241 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:35.349203 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:36.018919 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:36.018890 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:36.723659 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:36.723614 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:37.519527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:37.516901 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:37.864767 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:37.864742 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:38.004586 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:38.004556 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:38.293735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:38.293649 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:38.936202 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:38.936164 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:39.644071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:39.644034 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:40.346626 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:40.346495 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:41.029438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:41.029379 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:41.813573 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:41.813544 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:42.511251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:42.511216 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:43.154243 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:43.154204 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:44.077882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:44.077852 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:44.721795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:44.721764 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:45.348370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:45.348330 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:46.109602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:46.109556 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:46.830851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:46.830800 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:47.652082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:47.652043 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:48.287046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:48.287008 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:48.920387 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:48.920353 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:49.598248 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:49.598219 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:50.310970 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:50.310937 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:51.061186 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:51.056422 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:51.712570 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:51.712541 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:52.383962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:52.383880 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:53.051984 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:53.051954 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:53.781883 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:53.781846 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:54.431502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:54.431457 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:55.118493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:55.118256 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:55.794956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:55.794915 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:56.133432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:56.133400 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:57:56.438773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:56.438689 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:57.415416 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:57.415363 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:58.080936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:58.080895 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:58.785147 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:58.785116 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:59.542915 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:57:59.542880 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:00.267054 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:00.267010 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:00.343838 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:58:00.343805 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:58:00.951642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:00.951582 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:01.651041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:01.650997 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:02.345485 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:02.341987 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:03.022809 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:03.022775 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:03.752938 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:03.752898 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:04.434689 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:04.434637 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:05.152511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:05.152450 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:05.844484 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:05.842505 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:06.522037 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:06.522004 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:07.159533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:07.159498 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:07.919787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:07.919759 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:08.690980 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:08.690949 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:08.823066 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:08.823032 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:09.344978 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:09.344948 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:09.493663 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:09.492720 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:10.120323 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:10.120289 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:10.832288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:10.832251 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:11.540175 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:11.540144 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:12.239138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:12.239097 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:12.931345 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:12.931309 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:13.730432 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:13.730405 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:14.397849 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:14.397815 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:15.113508 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:15.113477 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:15.818334 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:15.818303 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:16.514639 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:16.514603 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:17.231182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:17.231150 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:17.918678 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:17.918615 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:18.640632 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:18.640592 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:19.447894 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:19.447854 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:20.242363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:20.242332 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:20.943557 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:20.943524 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:21.689892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:21.689856 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:22.535699 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:22.535669 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:23.222478 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:23.222440 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:23.875326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:23.875296 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:24.525320 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:24.525259 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:25.336085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:25.336055 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:26.037109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:26.037071 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:26.725558 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:26.725524 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:27.422172 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:27.422127 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:28.150935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:28.150895 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:28.830183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:28.830148 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:29.531656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:29.531617 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:30.243162 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:30.243116 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:30.343653 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:58:30.343616 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:58:31.062010 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:31.061928 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:31.835544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:31.835497 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:32.367702 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:32.367671 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb"] Oct 31 12:58:32.447835 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:32.447806 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:33.006967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:33.006934 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:33.737424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:33.737375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:34.300095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:34.300065 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt"] Oct 31 12:58:34.425351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:34.425176 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:34.701270 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:34.701242 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:35.128752 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:35.128719 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:35.647479 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:35.647429 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:36.412746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:36.412717 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:37.169029 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:37.168998 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:37.866096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:37.866044 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:38.614421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:38.614388 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:39.485983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:39.485930 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:40.248775 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:40.248743 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:40.948405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:40.944018 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:41.646647 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:41.646608 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:42.450406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:42.450309 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:43.153779 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:43.153731 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:43.569934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:43.569905 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp"] Oct 31 12:58:43.645677 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:43.645648 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:43.997550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:43.997413 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:44.849964 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:44.849925 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:45.528502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:45.528440 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:45.886341 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:45.886307 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:46.527148 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:46.527109 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:47.359807 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.359749 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:47.389095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.389035 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" podUID="888b8db5-5f7a-49f1-91a7-7ddabdd54f00" containerName="main" containerID="cri-o://cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4" gracePeriod=45 Oct 31 12:58:47.668138 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.668111 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:58:47.749651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.749622 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-dshm\") pod \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " Oct 31 12:58:47.749812 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.749686 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-model-cache\") pod \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " Oct 31 12:58:47.749896 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.749877 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-model-cache" pod "888b8db5-5f7a-49f1-91a7-7ddabdd54f00" (UID: "888b8db5-5f7a-49f1-91a7-7ddabdd54f00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:47.750009 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.749992 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tmp-dir\") pod \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " Oct 31 12:58:47.750115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750038 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-kube-api-access-72vbc\" (UniqueName: \"kubernetes.io/projected/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-kube-api-access-72vbc\") pod \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " Oct 31 12:58:47.750115 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750092 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tls-certs\" (UniqueName: \"kubernetes.io/secret/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tls-certs\") pod \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " Oct 31 12:58:47.750258 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750232 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-home\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-home\") pod \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\" (UID: \"888b8db5-5f7a-49f1-91a7-7ddabdd54f00\") " Oct 31 12:58:47.750331 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750232 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tmp-dir" pod "888b8db5-5f7a-49f1-91a7-7ddabdd54f00" (UID: "888b8db5-5f7a-49f1-91a7-7ddabdd54f00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:47.750493 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750446 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-home" pod "888b8db5-5f7a-49f1-91a7-7ddabdd54f00" (UID: "888b8db5-5f7a-49f1-91a7-7ddabdd54f00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:47.750831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750813 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:47.750831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750832 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:47.750990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.750841 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:47.751956 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.751935 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-dshm" pod "888b8db5-5f7a-49f1-91a7-7ddabdd54f00" (UID: "888b8db5-5f7a-49f1-91a7-7ddabdd54f00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:47.752247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.752229 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tls-certs" pod "888b8db5-5f7a-49f1-91a7-7ddabdd54f00" (UID: "888b8db5-5f7a-49f1-91a7-7ddabdd54f00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:47.752291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.752258 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-kube-api-access-72vbc" pod "888b8db5-5f7a-49f1-91a7-7ddabdd54f00" (UID: "888b8db5-5f7a-49f1-91a7-7ddabdd54f00"). InnerVolumeSpecName "kube-api-access-72vbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:47.852156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.852130 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vbc\" (UniqueName: \"kubernetes.io/projected/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-kube-api-access-72vbc\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:47.852156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.852154 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:47.852346 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:47.852164 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b8db5-5f7a-49f1-91a7-7ddabdd54f00-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:48.056330 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.047926 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:48.217259 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.217227 6727 generic.go:356] "Generic (PLEG): container finished" podID="888b8db5-5f7a-49f1-91a7-7ddabdd54f00" containerID="cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4" exitCode=0 Oct 31 12:58:48.217429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.217268 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" Oct 31 12:58:48.217429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.217310 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" event={"ID":"888b8db5-5f7a-49f1-91a7-7ddabdd54f00","Type":"ContainerDied","Data":"cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4"} Oct 31 12:58:48.217429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.217351 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb" event={"ID":"888b8db5-5f7a-49f1-91a7-7ddabdd54f00","Type":"ContainerDied","Data":"16c13008476331419de1fa4a5a11d43f26103cf700e0fcdf39c2522ac990bf9f"} Oct 31 12:58:48.217429 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.217368 6727 scope.go:122] "RemoveContainer" containerID="cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4" Oct 31 12:58:48.228513 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.228492 6727 scope.go:122] "RemoveContainer" containerID="cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4" Oct 31 12:58:48.228765 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:58:48.228748 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4\": container with ID starting with cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4 not found: ID does not exist" containerID="cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4" Oct 31 12:58:48.228847 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.228771 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4"} err="failed to get container status \"cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4\": rpc error: code = NotFound desc = could not find container \"cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4\": container with ID starting with cbba6191745c575b7bc192346debfcd598ff537ac88c951d346f8f7a0e6179c4 not found: ID does not exist" Oct 31 12:58:48.240933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.240912 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb"] Oct 31 12:58:48.243758 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.243738 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d086f3/test-model-d086f3-kserve-675466565-lw8qb"] Oct 31 12:58:48.823644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:48.823601 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:49.302358 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.302287 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888b8db5-5f7a-49f1-91a7-7ddabdd54f00" path="/var/lib/kubelet/pods/888b8db5-5f7a-49f1-91a7-7ddabdd54f00/volumes" Oct 31 12:58:49.320520 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.320490 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" podUID="c60a1c48-7c98-4f57-bda4-1d5aa6be1273" containerName="main" containerID="cri-o://5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d" gracePeriod=45 Oct 31 12:58:49.535704 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.535514 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:49.619643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.619619 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:58:49.684113 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684085 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tmp-dir\") pod \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " Oct 31 12:58:49.684251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684123 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tls-certs\") pod \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " Oct 31 12:58:49.684251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684210 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-dshm\") pod \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " Oct 31 12:58:49.684251 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684234 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-kube-api-access-vrdww\" (UniqueName: \"kubernetes.io/projected/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-kube-api-access-vrdww\") pod \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " Oct 31 12:58:49.684421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684269 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-model-cache\") pod \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " Oct 31 12:58:49.684421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684295 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-home\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-home\") pod \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\" (UID: \"c60a1c48-7c98-4f57-bda4-1d5aa6be1273\") " Oct 31 12:58:49.684421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684328 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tmp-dir" pod "c60a1c48-7c98-4f57-bda4-1d5aa6be1273" (UID: "c60a1c48-7c98-4f57-bda4-1d5aa6be1273"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:49.684636 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684604 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-home" pod "c60a1c48-7c98-4f57-bda4-1d5aa6be1273" (UID: "c60a1c48-7c98-4f57-bda4-1d5aa6be1273"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:49.684636 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684621 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-model-cache" pod "c60a1c48-7c98-4f57-bda4-1d5aa6be1273" (UID: "c60a1c48-7c98-4f57-bda4-1d5aa6be1273"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:49.684892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684871 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:49.684892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684892 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:49.685028 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.684907 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:49.686375 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.686351 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tls-certs" pod "c60a1c48-7c98-4f57-bda4-1d5aa6be1273" (UID: "c60a1c48-7c98-4f57-bda4-1d5aa6be1273"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:49.686496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.686378 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-dshm" pod "c60a1c48-7c98-4f57-bda4-1d5aa6be1273" (UID: "c60a1c48-7c98-4f57-bda4-1d5aa6be1273"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:49.686496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.686434 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-kube-api-access-vrdww" pod "c60a1c48-7c98-4f57-bda4-1d5aa6be1273" (UID: "c60a1c48-7c98-4f57-bda4-1d5aa6be1273"). InnerVolumeSpecName "kube-api-access-vrdww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:49.785651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.785622 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:49.785651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.785647 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:49.785824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:49.785661 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrdww\" (UniqueName: \"kubernetes.io/projected/c60a1c48-7c98-4f57-bda4-1d5aa6be1273-kube-api-access-vrdww\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.040882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.040749 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:50.139438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.139407 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:58:50.232257 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.232221 6727 generic.go:356] "Generic (PLEG): container finished" podID="c60a1c48-7c98-4f57-bda4-1d5aa6be1273" containerID="5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d" exitCode=0 Oct 31 12:58:50.232424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.232267 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" Oct 31 12:58:50.232424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.232313 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" event={"ID":"c60a1c48-7c98-4f57-bda4-1d5aa6be1273","Type":"ContainerDied","Data":"5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d"} Oct 31 12:58:50.232424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.232358 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt" event={"ID":"c60a1c48-7c98-4f57-bda4-1d5aa6be1273","Type":"ContainerDied","Data":"5a07ea8c28fa35ef90886eaf1b2d493c615ae6f5df2d56283411b014572cefb8"} Oct 31 12:58:50.232424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.232378 6727 scope.go:122] "RemoveContainer" containerID="5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d" Oct 31 12:58:50.242977 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.242959 6727 scope.go:122] "RemoveContainer" containerID="5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d" Oct 31 12:58:50.243216 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:58:50.243198 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d\": container with ID starting with 5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d not found: ID does not exist" containerID="5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d" Oct 31 12:58:50.243293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.243227 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d"} err="failed to get container status \"5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d\": rpc error: code = NotFound desc = could not find container \"5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d\": container with ID starting with 5ca9c3d4f6139ccdd60a7c3c3738c296f1dfeca04b3970ae13e6192ed6e9069d not found: ID does not exist" Oct 31 12:58:50.263914 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.263893 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt"] Oct 31 12:58:50.288814 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.288786 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9b62a0/test-model-9b62a0-kserve-7698b68fb7-28ljt"] Oct 31 12:58:50.726745 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:50.726704 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:51.344498 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:51.343335 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60a1c48-7c98-4f57-bda4-1d5aa6be1273" path="/var/lib/kubelet/pods/c60a1c48-7c98-4f57-bda4-1d5aa6be1273/volumes" Oct 31 12:58:51.842932 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:51.842891 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:52.829784 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:52.829750 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:53.644805 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:53.644770 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:54.002840 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:54.002751 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg"] Oct 31 12:58:54.119126 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:54.119096 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:54.342945 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:54.342913 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:54.515864 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:54.515833 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:55.268026 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:55.267991 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:55.975672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:55.975575 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:56.138819 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:56.138780 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79"] Oct 31 12:58:56.148952 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:56.148924 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5"] Oct 31 12:58:56.250427 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:56.250350 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:56.392286 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:56.392254 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:56.707223 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:56.707130 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:57.574834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:57.574798 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:58.225979 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:58.225948 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:58.593050 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:58.592984 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" podUID="b0076d52-4c70-4355-b8f1-7c90781701b6" containerName="main" containerID="cri-o://0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060" gracePeriod=45 Oct 31 12:58:58.818363 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:58.818306 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:58.889828 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:58.889804 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:58:59.010487 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.008127 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" podUID="dd401d12-5957-4561-93ec-71b6a56e614b" containerName="payload-processing" containerID="cri-o://d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14" gracePeriod=25 Oct 31 12:58:59.033489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.031975 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:59.054629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.053772 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-tmp-dir\") pod \"b0076d52-4c70-4355-b8f1-7c90781701b6\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " Oct 31 12:58:59.054629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.053898 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0076d52-4c70-4355-b8f1-7c90781701b6-kube-api-access-d5t9v\" (UniqueName: \"kubernetes.io/projected/b0076d52-4c70-4355-b8f1-7c90781701b6-kube-api-access-d5t9v\") pod \"b0076d52-4c70-4355-b8f1-7c90781701b6\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " Oct 31 12:58:59.054629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.053999 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0076d52-4c70-4355-b8f1-7c90781701b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0076d52-4c70-4355-b8f1-7c90781701b6-tls-certs\") pod \"b0076d52-4c70-4355-b8f1-7c90781701b6\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " Oct 31 12:58:59.054629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.054098 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-model-cache\") pod \"b0076d52-4c70-4355-b8f1-7c90781701b6\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " Oct 31 12:58:59.054629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.054148 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-dshm\") pod \"b0076d52-4c70-4355-b8f1-7c90781701b6\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " Oct 31 12:58:59.054629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.054254 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-home\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-home\") pod \"b0076d52-4c70-4355-b8f1-7c90781701b6\" (UID: \"b0076d52-4c70-4355-b8f1-7c90781701b6\") " Oct 31 12:58:59.057130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.056847 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-tmp-dir" pod "b0076d52-4c70-4355-b8f1-7c90781701b6" (UID: "b0076d52-4c70-4355-b8f1-7c90781701b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:59.057130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.057100 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-home" pod "b0076d52-4c70-4355-b8f1-7c90781701b6" (UID: "b0076d52-4c70-4355-b8f1-7c90781701b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:59.057503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.057452 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-model-cache" pod "b0076d52-4c70-4355-b8f1-7c90781701b6" (UID: "b0076d52-4c70-4355-b8f1-7c90781701b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:59.078503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.077041 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0076d52-4c70-4355-b8f1-7c90781701b6-tls-certs" pod "b0076d52-4c70-4355-b8f1-7c90781701b6" (UID: "b0076d52-4c70-4355-b8f1-7c90781701b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:59.078503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.077161 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:59.078503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.077176 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:59.078503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.077184 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:59.082612 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.082561 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0076d52-4c70-4355-b8f1-7c90781701b6-kube-api-access-d5t9v" pod "b0076d52-4c70-4355-b8f1-7c90781701b6" (UID: "b0076d52-4c70-4355-b8f1-7c90781701b6"). InnerVolumeSpecName "kube-api-access-d5t9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:59.082728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.082545 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-dshm" pod "b0076d52-4c70-4355-b8f1-7c90781701b6" (UID: "b0076d52-4c70-4355-b8f1-7c90781701b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:59.179081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.179009 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5t9v\" (UniqueName: \"kubernetes.io/projected/b0076d52-4c70-4355-b8f1-7c90781701b6-kube-api-access-d5t9v\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:59.179081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.179046 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0076d52-4c70-4355-b8f1-7c90781701b6-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:59.179081 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.179062 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0076d52-4c70-4355-b8f1-7c90781701b6-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:59.183870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.183844 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:59.311364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.311325 6727 generic.go:356] "Generic (PLEG): container finished" podID="b0076d52-4c70-4355-b8f1-7c90781701b6" containerID="0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060" exitCode=0 Oct 31 12:58:59.311580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.311374 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" Oct 31 12:58:59.311580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.311412 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" event={"ID":"b0076d52-4c70-4355-b8f1-7c90781701b6","Type":"ContainerDied","Data":"0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060"} Oct 31 12:58:59.311580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.311483 6727 scope.go:122] "RemoveContainer" containerID="0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060" Oct 31 12:58:59.311580 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.311507 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp" event={"ID":"b0076d52-4c70-4355-b8f1-7c90781701b6","Type":"ContainerDied","Data":"17bc8bf75999b322c894487fcddb64f42d71382c6d39065c0c48524bdfba70e5"} Oct 31 12:58:59.317060 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.317035 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:59.322288 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.322270 6727 scope.go:122] "RemoveContainer" containerID="0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060" Oct 31 12:58:59.322588 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:58:59.322568 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060\": container with ID starting with 0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060 not found: ID does not exist" containerID="0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060" Oct 31 12:58:59.322656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.322600 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060"} err="failed to get container status \"0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060\": rpc error: code = NotFound desc = could not find container \"0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060\": container with ID starting with 0467f642bacb454c111f2749479789409e937e89f46add7b982ea707c9d0f060 not found: ID does not exist" Oct 31 12:58:59.343399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.343372 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp"] Oct 31 12:58:59.346135 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.346108 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-71bc05/test-model-71bc05-kserve-f5475b959-64jzp"] Oct 31 12:58:59.732159 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:58:59.732127 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:00.082716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:00.081544 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:00.345631 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:00.345547 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:59:00.382449 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:00.382411 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:00.580534 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:00.580501 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:01.021180 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:01.021128 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:01.139891 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:01.139858 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" podUID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" containerName="payload-processing" containerID="cri-o://7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3" gracePeriod=25 Oct 31 12:59:01.149342 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:01.149313 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" podUID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" containerName="payload-pre-processing" containerID="cri-o://7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2" gracePeriod=25 Oct 31 12:59:01.301602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:01.301540 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0076d52-4c70-4355-b8f1-7c90781701b6" path="/var/lib/kubelet/pods/b0076d52-4c70-4355-b8f1-7c90781701b6/volumes" Oct 31 12:59:01.645244 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:01.645218 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:01.829305 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:01.829273 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:01.984708 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:01.984625 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:02.124227 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:02.124192 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:02.440194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:02.440167 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:02.740271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:02.740010 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:03.521396 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:03.521368 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:04.350579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:04.350535 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:05.070272 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:05.070230 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:05.809843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:05.809811 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:06.485886 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:06.485856 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:06.922820 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:06.922706 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:07.051228 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:07.051194 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:07.157274 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:07.157241 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn"] Oct 31 12:59:07.157574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:07.157546 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" containerID="cri-o://dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103" gracePeriod=30 Oct 31 12:59:07.200531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:07.200439 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:07.338302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:07.338269 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:07.738519 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:07.738452 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" podUID="dd401d12-5957-4561-93ec-71b6a56e614b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:07.994808 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:07.994643 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:08.804312 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:08.804280 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:08.981489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:08.973828 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:09.119993 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:09.119964 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:09.335316 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:09.335283 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:09.585296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:09.585256 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:09.755151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:09.755118 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:10.063350 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:10.063247 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:10.085451 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:10.085421 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb"] Oct 31 12:59:10.085692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:10.085669 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" containerID="cri-o://f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60" gracePeriod=30 Oct 31 12:59:10.463364 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:10.463330 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" podUID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:10.463851 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:10.463823 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" podUID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:10.832533 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:10.832458 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:11.004500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.004437 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:11.239577 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.239492 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb"] Oct 31 12:59:11.240063 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.239726 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" containerID="cri-o://64f40430fcf60519d4442a051d8b546dfc5611a555366753e4d7a1f4d58ce28d" gracePeriod=30 Oct 31 12:59:11.259609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.259574 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf"] Oct 31 12:59:11.259771 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.259752 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" containerID="cri-o://f6c60dbf774bce16593610611bfbde684d7ca47d3ee47d9f1530f09d7ebde4d1" gracePeriod=30 Oct 31 12:59:11.262415 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.262392 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp"] Oct 31 12:59:11.262627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.262608 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" containerID="cri-o://abbc0dfc7fc9e08e8f574531ed5879116f9307ad40743a52cde8a437911f82bb" gracePeriod=30 Oct 31 12:59:11.339644 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.339605 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:11.633137 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:11.633106 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:12.445863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.445838 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:59:12.450404 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.450370 6727 generic.go:356] "Generic (PLEG): container finished" podID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerID="dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103" exitCode=0 Oct 31 12:59:12.450527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.450442 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" event={"ID":"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa","Type":"ContainerDied","Data":"dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103"} Oct 31 12:59:12.450527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.450515 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" event={"ID":"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa","Type":"ContainerDied","Data":"2259b46b083fa4af1ab109d30742397fe26a920b9c51c460260dbf33595cbee2"} Oct 31 12:59:12.450607 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.450539 6727 scope.go:122] "RemoveContainer" containerID="dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103" Oct 31 12:59:12.460356 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.460336 6727 scope.go:122] "RemoveContainer" containerID="baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a" Oct 31 12:59:12.470413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.470391 6727 scope.go:122] "RemoveContainer" containerID="dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103" Oct 31 12:59:12.470743 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:12.470719 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103\": container with ID starting with dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103 not found: ID does not exist" containerID="dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103" Oct 31 12:59:12.470838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.470753 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103"} err="failed to get container status \"dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103\": rpc error: code = NotFound desc = could not find container \"dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103\": container with ID starting with dfd7a5d931ff7ebe0e3231420b668ac561184251732fb26c66615ec5d08f5103 not found: ID does not exist" Oct 31 12:59:12.470838 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.470770 6727 scope.go:122] "RemoveContainer" containerID="baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a" Oct 31 12:59:12.471070 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:12.471052 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a\": container with ID starting with baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a not found: ID does not exist" containerID="baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a" Oct 31 12:59:12.471151 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.471075 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a"} err="failed to get container status \"baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a\": rpc error: code = NotFound desc = could not find container \"baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a\": container with ID starting with baeb45c536ed7d36afbfa30ad0fdf875be10e9689563b451544f6d75339e806a not found: ID does not exist" Oct 31 12:59:12.512550 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.512491 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:12.519853 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.519821 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-envoy\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.520014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.519874 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-kube-api-access-ttzlj\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-kube-api-access-ttzlj\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.520014 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.520007 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-credential-socket\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.520155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.520064 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-certs\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.520155 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.520150 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-token\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-token\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.520267 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.520185 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-podinfo\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.520267 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.520244 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istiod-ca-cert\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.520366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.520347 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-data\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.521309 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.521283 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-certs" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:12.521603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.521577 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istiod-ca-cert" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:12.521694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.521658 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-credential-socket" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:12.521955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.521933 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-data" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:12.523055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.521949 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-socket" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:12.523234 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.522790 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-socket\") pod \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\" (UID: \"80dd643e-ae00-46a6-9d8a-a7f8aaf636aa\") " Oct 31 12:59:12.527553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.527524 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.527670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.527557 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.527670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.527572 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.527670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.527585 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.527670 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.527600 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.532016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.531983 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-podinfo" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:12.532114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.532010 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-envoy" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:12.532114 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.532012 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-token" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:12.534629 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.533821 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-kube-api-access-ttzlj" pod "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" (UID: "80dd643e-ae00-46a6-9d8a-a7f8aaf636aa"). InnerVolumeSpecName "kube-api-access-ttzlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:12.628842 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.628819 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.628947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.628845 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.628947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.628855 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:12.628947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:12.628865 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzlj\" (UniqueName: \"kubernetes.io/projected/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa-kube-api-access-ttzlj\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:13.199142 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:13.199098 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:13.457816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:13.457758 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn" Oct 31 12:59:13.479762 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:13.479734 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn"] Oct 31 12:59:13.485661 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:13.485636 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-b9rdn"] Oct 31 12:59:13.680354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:13.680309 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:13.818239 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:13.818173 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:14.009858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:14.009833 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:14.493407 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:14.493379 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:59:14.493789 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:14.493438 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:59:14.819934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:14.819901 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:15.304343 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.304308 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" path="/var/lib/kubelet/pods/80dd643e-ae00-46a6-9d8a-a7f8aaf636aa/volumes" Oct 31 12:59:15.347795 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.347737 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:59:15.463503 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463451 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-kube-api-access-q76ws\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-kube-api-access-q76ws\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.463671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463536 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-socket\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.463671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463584 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-certs\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.463671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463602 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istiod-ca-cert\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.463671 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463651 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-podinfo\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.463947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463761 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-credential-socket\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.463947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463842 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-envoy\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.463947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463860 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-certs" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:15.463947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463875 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-socket" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:15.463947 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463898 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-data\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.464185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.463948 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-token\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-token\") pod \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\" (UID: \"2faa2b8a-979b-4b84-a35a-32d9b7413e75\") " Oct 31 12:59:15.464185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464018 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-credential-socket" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:15.464185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464135 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-data" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:15.464185 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464062 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istiod-ca-cert" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:15.464709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464600 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.464709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464632 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.464709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464646 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.464709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464660 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.464709 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.464674 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.465990 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.465961 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-kube-api-access-q76ws" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "kube-api-access-q76ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:15.466294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.466268 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-podinfo" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:15.466447 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.466424 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-envoy" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:15.466574 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.466549 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-token" pod "2faa2b8a-979b-4b84-a35a-32d9b7413e75" (UID: "2faa2b8a-979b-4b84-a35a-32d9b7413e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:15.472187 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.472165 6727 generic.go:356] "Generic (PLEG): container finished" podID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerID="f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60" exitCode=0 Oct 31 12:59:15.472296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.472209 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" Oct 31 12:59:15.472411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.472211 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" event={"ID":"2faa2b8a-979b-4b84-a35a-32d9b7413e75","Type":"ContainerDied","Data":"f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60"} Oct 31 12:59:15.472411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.472339 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb" event={"ID":"2faa2b8a-979b-4b84-a35a-32d9b7413e75","Type":"ContainerDied","Data":"217d5298d608dd3e01ceb8c242d317da113efc427e39b1ea978f076b4d93dc73"} Oct 31 12:59:15.472411 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.472371 6727 scope.go:122] "RemoveContainer" containerID="f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60" Oct 31 12:59:15.499289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.499267 6727 scope.go:122] "RemoveContainer" containerID="f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60" Oct 31 12:59:15.499623 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:15.499549 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60\": container with ID starting with f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60 not found: ID does not exist" containerID="f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60" Oct 31 12:59:15.499623 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.499580 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60"} err="failed to get container status \"f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60\": rpc error: code = NotFound desc = could not find container \"f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60\": container with ID starting with f8f5ea56921a541bce542dc27365604bbb9985ed8c143f03e6c4d67363fe6d60 not found: ID does not exist" Oct 31 12:59:15.531710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.531687 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb"] Oct 31 12:59:15.535931 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.535910 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-267gb"] Oct 31 12:59:15.566473 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.566432 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.566473 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.566457 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.566639 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.566487 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.566639 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.566501 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q76ws\" (UniqueName: \"kubernetes.io/projected/2faa2b8a-979b-4b84-a35a-32d9b7413e75-kube-api-access-q76ws\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:15.666125 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.666008 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:59:15.666125 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.666100 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:59:15.672434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.672404 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:59:15.672627 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.672453 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:59:15.862378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:15.862330 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:16.486211 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.486182 6727 generic.go:356] "Generic (PLEG): container finished" podID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerID="f6c60dbf774bce16593610611bfbde684d7ca47d3ee47d9f1530f09d7ebde4d1" exitCode=0 Oct 31 12:59:16.486390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.486274 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" event={"ID":"f45cda06-0d2f-45f4-917f-e854e4cb30ba","Type":"ContainerDied","Data":"f6c60dbf774bce16593610611bfbde684d7ca47d3ee47d9f1530f09d7ebde4d1"} Oct 31 12:59:16.488289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.488260 6727 generic.go:356] "Generic (PLEG): container finished" podID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerID="abbc0dfc7fc9e08e8f574531ed5879116f9307ad40743a52cde8a437911f82bb" exitCode=0 Oct 31 12:59:16.488421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.488383 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" event={"ID":"62233c53-e636-4b2d-b52f-2b9959ee47fd","Type":"ContainerDied","Data":"abbc0dfc7fc9e08e8f574531ed5879116f9307ad40743a52cde8a437911f82bb"} Oct 31 12:59:16.491735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.491709 6727 generic.go:356] "Generic (PLEG): container finished" podID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerID="64f40430fcf60519d4442a051d8b546dfc5611a555366753e4d7a1f4d58ce28d" exitCode=0 Oct 31 12:59:16.491863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.491780 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" event={"ID":"7c8e0968-6f24-450b-8f8f-e7b00f1fb768","Type":"ContainerDied","Data":"64f40430fcf60519d4442a051d8b546dfc5611a555366753e4d7a1f4d58ce28d"} Oct 31 12:59:16.573855 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.573823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:16.612922 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.612894 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:59:16.614870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.614849 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:59:16.630215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.630197 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:59:16.682397 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.682362 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-podinfo\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.682585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.682446 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62233c53-e636-4b2d-b52f-2b9959ee47fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62233c53-e636-4b2d-b52f-2b9959ee47fd-istiod-ca-cert\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.682869 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.682843 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62233c53-e636-4b2d-b52f-2b9959ee47fd-istiod-ca-cert" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:16.682953 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.682936 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-credential-socket\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.683012 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.682968 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-socket\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.683071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683029 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-certs\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.683071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683058 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-socket\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.683182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683105 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-token\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-token\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.683182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683164 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-kube-api-access-4fqnv\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-kube-api-access-4fqnv\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.683299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683218 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istiod-ca-cert\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.683299 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683260 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-data\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.683405 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683323 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-socket\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.683480 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683403 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-certs" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.683929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683629 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-credential-socket" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.683929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683690 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-socket" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.683929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683701 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-socket" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.683929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683831 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-data" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.683929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683891 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istiod-ca-cert" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:16.683929 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.683999 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-socket" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.684400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684368 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-token\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-token\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.684454 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684407 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-token\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-token\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.684537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684476 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-envoy\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.684537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684507 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-certs\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.684643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684533 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-envoy\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.684643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684572 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-kube-api-access-2xmw5\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-kube-api-access-2xmw5\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.684643 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684618 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-kube-api-access-pkqbz\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-kube-api-access-pkqbz\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.684797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684643 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istiod-ca-cert\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.684797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684695 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-podinfo\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.684797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684726 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-data\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.684797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684758 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-credential-socket\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.684797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684791 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-certs\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.685055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.684951 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-podinfo" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:16.685164 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.685139 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-data\") pod \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\" (UID: \"7c8e0968-6f24-450b-8f8f-e7b00f1fb768\") " Oct 31 12:59:16.685324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.685181 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-envoy\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.685324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.685228 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-credential-socket\") pod \"62233c53-e636-4b2d-b52f-2b9959ee47fd\" (UID: \"62233c53-e636-4b2d-b52f-2b9959ee47fd\") " Oct 31 12:59:16.685489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.685318 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-data" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.685489 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.685389 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-podinfo\") pod \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\" (UID: \"f45cda06-0d2f-45f4-917f-e854e4cb30ba\") " Oct 31 12:59:16.685972 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.685945 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-token" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:16.686421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.686173 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istiod-ca-cert" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:16.686421 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.686380 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-data" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.686603 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.686478 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-credential-socket" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.686665 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.686642 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-certs" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687126 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687153 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62233c53-e636-4b2d-b52f-2b9959ee47fd-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687169 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687171 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-certs" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687184 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687219 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687237 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687253 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687279 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687292 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687308 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687322 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.687360 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687359 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.688055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687373 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.688055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687387 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.688055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.687400 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.688291 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.688259 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-credential-socket" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.694046 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.694003 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-token" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:16.694166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.694053 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-token" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:16.694166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.694157 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-envoy" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.694634 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.694556 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-envoy" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.695276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.695182 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-kube-api-access-2xmw5" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "kube-api-access-2xmw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:16.695787 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.695306 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-kube-api-access-4fqnv" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "kube-api-access-4fqnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:16.696196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.694242 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-kube-api-access-pkqbz" pod "7c8e0968-6f24-450b-8f8f-e7b00f1fb768" (UID: "7c8e0968-6f24-450b-8f8f-e7b00f1fb768"). InnerVolumeSpecName "kube-api-access-pkqbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:16.697121 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.697094 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-podinfo" pod "62233c53-e636-4b2d-b52f-2b9959ee47fd" (UID: "62233c53-e636-4b2d-b52f-2b9959ee47fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:16.698321 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.698294 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-podinfo" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:16.699439 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.699418 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-envoy" pod "f45cda06-0d2f-45f4-917f-e854e4cb30ba" (UID: "f45cda06-0d2f-45f4-917f-e854e4cb30ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:16.789326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789242 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fqnv\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-kube-api-access-4fqnv\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789277 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789292 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789305 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789326 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789319 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789331 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789344 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmw5\" (UniqueName: \"kubernetes.io/projected/62233c53-e636-4b2d-b52f-2b9959ee47fd-kube-api-access-2xmw5\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789357 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqbz\" (UniqueName: \"kubernetes.io/projected/7c8e0968-6f24-450b-8f8f-e7b00f1fb768-kube-api-access-pkqbz\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789370 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789383 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789396 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62233c53-e636-4b2d-b52f-2b9959ee47fd-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:16.789710 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:16.789409 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45cda06-0d2f-45f4-917f-e854e4cb30ba-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:17.150294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.150259 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:17.302199 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.302163 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" path="/var/lib/kubelet/pods/2faa2b8a-979b-4b84-a35a-32d9b7413e75/volumes" Oct 31 12:59:17.304691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.304661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:17.421802 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.421660 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8"] Oct 31 12:59:17.421967 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.421929 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" containerID="cri-o://cd81376acc6d238b83d584633c09a56a6e7429815f9dec7b6e4e164eab3da88f" gracePeriod=30 Oct 31 12:59:17.489690 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.489655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:17.524481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.517339 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" event={"ID":"f45cda06-0d2f-45f4-917f-e854e4cb30ba","Type":"ContainerDied","Data":"6bcd2fd63e0ad69fee4aa358fabe25edcb231007283b567a93b83c3f498b6652"} Oct 31 12:59:17.524481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.517455 6727 scope.go:122] "RemoveContainer" containerID="f6c60dbf774bce16593610611bfbde684d7ca47d3ee47d9f1530f09d7ebde4d1" Oct 31 12:59:17.524481 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.517682 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf" Oct 31 12:59:17.529022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.528993 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" event={"ID":"62233c53-e636-4b2d-b52f-2b9959ee47fd","Type":"ContainerDied","Data":"ced9ed90a02e2e3214d025858af718399dd2a90ca78c77d2dc247561c1748e61"} Oct 31 12:59:17.529255 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.529238 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp" Oct 31 12:59:17.537537 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.537514 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" event={"ID":"7c8e0968-6f24-450b-8f8f-e7b00f1fb768","Type":"ContainerDied","Data":"23ee615d6f45902f9b95eda60c8c324a3cd224944991177ee5ef7acf8964c04b"} Oct 31 12:59:17.537982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.537834 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb" Oct 31 12:59:17.591048 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.591011 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf"] Oct 31 12:59:17.594217 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.594190 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-v65gf"] Oct 31 12:59:17.597834 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.597246 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp"] Oct 31 12:59:17.600279 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.600252 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4cnhp"] Oct 31 12:59:17.603297 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.603278 6727 scope.go:122] "RemoveContainer" containerID="abbc0dfc7fc9e08e8f574531ed5879116f9307ad40743a52cde8a437911f82bb" Oct 31 12:59:17.614799 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.614777 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb"] Oct 31 12:59:17.617399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.617376 6727 scope.go:122] "RemoveContainer" containerID="64f40430fcf60519d4442a051d8b546dfc5611a555366753e4d7a1f4d58ce28d" Oct 31 12:59:17.620620 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.620598 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-dk4jb"] Oct 31 12:59:17.638256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.637657 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:17.738264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.738190 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" podUID="dd401d12-5957-4561-93ec-71b6a56e614b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:17.975005 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:17.974968 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:18.128641 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:18.128612 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:18.643156 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:18.643125 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:19.218030 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.218000 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:59:19.302117 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.302028 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" path="/var/lib/kubelet/pods/62233c53-e636-4b2d-b52f-2b9959ee47fd/volumes" Oct 31 12:59:19.302575 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.302561 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" path="/var/lib/kubelet/pods/7c8e0968-6f24-450b-8f8f-e7b00f1fb768/volumes" Oct 31 12:59:19.302998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.302984 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" path="/var/lib/kubelet/pods/f45cda06-0d2f-45f4-917f-e854e4cb30ba/volumes" Oct 31 12:59:19.343370 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.343347 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd401d12-5957-4561-93ec-71b6a56e614b-kube-api-access-twzbm\" (UniqueName: \"kubernetes.io/projected/dd401d12-5957-4561-93ec-71b6a56e614b-kube-api-access-twzbm\") pod \"dd401d12-5957-4561-93ec-71b6a56e614b\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " Oct 31 12:59:19.343514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.343386 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd401d12-5957-4561-93ec-71b6a56e614b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd401d12-5957-4561-93ec-71b6a56e614b-plugins-config-volume\") pod \"dd401d12-5957-4561-93ec-71b6a56e614b\" (UID: \"dd401d12-5957-4561-93ec-71b6a56e614b\") " Oct 31 12:59:19.343789 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.343763 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd401d12-5957-4561-93ec-71b6a56e614b-plugins-config-volume" pod "dd401d12-5957-4561-93ec-71b6a56e614b" (UID: "dd401d12-5957-4561-93ec-71b6a56e614b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:19.344071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.344050 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd401d12-5957-4561-93ec-71b6a56e614b-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:19.345548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.345524 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd401d12-5957-4561-93ec-71b6a56e614b-kube-api-access-twzbm" pod "dd401d12-5957-4561-93ec-71b6a56e614b" (UID: "dd401d12-5957-4561-93ec-71b6a56e614b"). InnerVolumeSpecName "kube-api-access-twzbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:19.445591 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.445567 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twzbm\" (UniqueName: \"kubernetes.io/projected/dd401d12-5957-4561-93ec-71b6a56e614b-kube-api-access-twzbm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:19.456551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.456528 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:19.556071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.555975 6727 generic.go:356] "Generic (PLEG): container finished" podID="dd401d12-5957-4561-93ec-71b6a56e614b" containerID="d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14" exitCode=0 Oct 31 12:59:19.556071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.556025 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" Oct 31 12:59:19.556071 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.556052 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" event={"ID":"dd401d12-5957-4561-93ec-71b6a56e614b","Type":"ContainerDied","Data":"d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14"} Oct 31 12:59:19.556351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.556116 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg" event={"ID":"dd401d12-5957-4561-93ec-71b6a56e614b","Type":"ContainerDied","Data":"39f28b35e4210ca4c4e33020d336a64960331e1d970fab9c0230e63bbf2fc337"} Oct 31 12:59:19.556351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.556148 6727 scope.go:122] "RemoveContainer" containerID="d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14" Oct 31 12:59:19.566672 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.566653 6727 scope.go:122] "RemoveContainer" containerID="d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14" Oct 31 12:59:19.566956 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:19.566934 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14\": container with ID starting with d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14 not found: ID does not exist" containerID="d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14" Oct 31 12:59:19.566998 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.566965 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14"} err="failed to get container status \"d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14\": rpc error: code = NotFound desc = could not find container \"d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14\": container with ID starting with d2011536c9bd9885ce92235e8da207de761916bb41a3b3395829ac76099d0d14 not found: ID does not exist" Oct 31 12:59:19.583191 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.583160 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg"] Oct 31 12:59:19.585654 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:19.585636 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d086f3-758bd8d747-z4vgg"] Oct 31 12:59:20.157027 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:20.157001 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:20.341593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:20.341499 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:20.464183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:20.464103 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" podUID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:20.464183 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:20.464147 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" podUID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:20.494229 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:20.494203 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:20.636024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:20.635994 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:20.910625 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:20.910567 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:21.218290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.218255 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:21.302386 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.302351 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd401d12-5957-4561-93ec-71b6a56e614b" path="/var/lib/kubelet/pods/dd401d12-5957-4561-93ec-71b6a56e614b/volumes" Oct 31 12:59:21.434194 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.434139 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:59:21.455936 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.455914 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:59:21.573526 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.573500 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5a92fb-09f8-4caf-8a18-652b94bfa460-kube-api-access-8bxlf\" (UniqueName: \"kubernetes.io/projected/3c5a92fb-09f8-4caf-8a18-652b94bfa460-kube-api-access-8bxlf\") pod \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " Oct 31 12:59:21.573684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.573555 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-plugins-config-volume\") pod \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " Oct 31 12:59:21.573684 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.573654 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5a92fb-09f8-4caf-8a18-652b94bfa460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5a92fb-09f8-4caf-8a18-652b94bfa460-plugins-config-volume\") pod \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\" (UID: \"3c5a92fb-09f8-4caf-8a18-652b94bfa460\") " Oct 31 12:59:21.573827 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.573689 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-kube-api-access-nxb6b\" (UniqueName: \"kubernetes.io/projected/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-kube-api-access-nxb6b\") pod \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\" (UID: \"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa\") " Oct 31 12:59:21.573923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.573895 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-plugins-config-volume" pod "4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" (UID: "4fdb1c5f-7bad-47ae-a852-e892a74f5cfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:21.574085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574035 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5a92fb-09f8-4caf-8a18-652b94bfa460-plugins-config-volume" pod "3c5a92fb-09f8-4caf-8a18-652b94bfa460" (UID: "3c5a92fb-09f8-4caf-8a18-652b94bfa460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:21.574354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574288 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:21.574354 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574314 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5a92fb-09f8-4caf-8a18-652b94bfa460-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:21.574610 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574584 6727 generic.go:356] "Generic (PLEG): container finished" podID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" containerID="7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3" exitCode=0 Oct 31 12:59:21.574676 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574644 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" Oct 31 12:59:21.574743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574672 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" event={"ID":"3c5a92fb-09f8-4caf-8a18-652b94bfa460","Type":"ContainerDied","Data":"7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3"} Oct 31 12:59:21.574800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574748 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79" event={"ID":"3c5a92fb-09f8-4caf-8a18-652b94bfa460","Type":"ContainerDied","Data":"b85ef06c9df04dac142027d6f09e9d3d0a0ed478d34b71849ca6d74fe4ba66eb"} Oct 31 12:59:21.574800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.574783 6727 scope.go:122] "RemoveContainer" containerID="7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3" Oct 31 12:59:21.576174 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.576150 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5a92fb-09f8-4caf-8a18-652b94bfa460-kube-api-access-8bxlf" pod "3c5a92fb-09f8-4caf-8a18-652b94bfa460" (UID: "3c5a92fb-09f8-4caf-8a18-652b94bfa460"). InnerVolumeSpecName "kube-api-access-8bxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:21.576276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.576189 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-kube-api-access-nxb6b" pod "4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" (UID: "4fdb1c5f-7bad-47ae-a852-e892a74f5cfa"). InnerVolumeSpecName "kube-api-access-nxb6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:21.576276 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.576234 6727 generic.go:356] "Generic (PLEG): container finished" podID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" containerID="7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2" exitCode=0 Oct 31 12:59:21.576413 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.576386 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" event={"ID":"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa","Type":"ContainerDied","Data":"7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2"} Oct 31 12:59:21.576536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.576436 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" event={"ID":"4fdb1c5f-7bad-47ae-a852-e892a74f5cfa","Type":"ContainerDied","Data":"05aa8e70a410a2dbab9edc61349728dab0506a6a7bac8d0310513d8bbf881b18"} Oct 31 12:59:21.576536 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.576448 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5" Oct 31 12:59:21.586339 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.586318 6727 scope.go:122] "RemoveContainer" containerID="7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3" Oct 31 12:59:21.586653 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:21.586630 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3\": container with ID starting with 7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3 not found: ID does not exist" containerID="7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3" Oct 31 12:59:21.586754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.586664 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3"} err="failed to get container status \"7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3\": rpc error: code = NotFound desc = could not find container \"7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3\": container with ID starting with 7205682d5148df971b794d93a36dc39871b7d059528370e9fc37b0c5d76fc0a3 not found: ID does not exist" Oct 31 12:59:21.586754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.586686 6727 scope.go:122] "RemoveContainer" containerID="7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2" Oct 31 12:59:21.595022 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.595002 6727 scope.go:122] "RemoveContainer" containerID="7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2" Oct 31 12:59:21.595274 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:21.595241 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2\": container with ID starting with 7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2 not found: ID does not exist" containerID="7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2" Oct 31 12:59:21.595332 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.595281 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2"} err="failed to get container status \"7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2\": rpc error: code = NotFound desc = could not find container \"7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2\": container with ID starting with 7bc33580a0b63f84177aff7f2db158817fecb720d9b56e0a52ef042bca376ad2 not found: ID does not exist" Oct 31 12:59:21.608862 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.608832 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5"] Oct 31 12:59:21.611823 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.611804 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b62a0-5476645bdc-j9mg5"] Oct 31 12:59:21.675962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.675940 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bxlf\" (UniqueName: \"kubernetes.io/projected/3c5a92fb-09f8-4caf-8a18-652b94bfa460-kube-api-access-8bxlf\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:21.675962 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.675962 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxb6b\" (UniqueName: \"kubernetes.io/projected/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa-kube-api-access-nxb6b\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:21.896105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.896073 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6"] Oct 31 12:59:21.910617 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.910590 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79"] Oct 31 12:59:21.914816 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.914788 6727 patch_prober.go:29] interesting pod/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:59:21.914969 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.914839 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:59:21.920642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.920486 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b62a0-69d46cc745-7mg79"] Oct 31 12:59:21.937008 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:21.936979 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:22.092327 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.092301 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:22.585986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.585954 6727 generic.go:356] "Generic (PLEG): container finished" podID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerID="cd81376acc6d238b83d584633c09a56a6e7429815f9dec7b6e4e164eab3da88f" exitCode=0 Oct 31 12:59:22.586760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.586042 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" event={"ID":"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d","Type":"ContainerDied","Data":"cd81376acc6d238b83d584633c09a56a6e7429815f9dec7b6e4e164eab3da88f"} Oct 31 12:59:22.586760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.586130 6727 scope.go:122] "RemoveContainer" containerID="475c06dc335837dbd5890eed6c40fd6c7a69a2c25d26216fe9103d845df196f3" Oct 31 12:59:22.650453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.641661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:22.716667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.716646 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:59:22.791215 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791192 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-data\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791382 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791275 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-envoy\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791453 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791414 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istiod-ca-cert\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791452 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-podinfo\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791509 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-data" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:22.791642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791540 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-token\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-token\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791642 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791577 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-credential-socket\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791637 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-kube-api-access-wfjzs\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-kube-api-access-wfjzs\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791746 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791712 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-certs\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791856 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791758 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-socket\") pod \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\" (UID: \"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d\") " Oct 31 12:59:22.791966 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.791850 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istiod-ca-cert" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:22.792175 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.792151 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-socket" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:22.792296 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.792272 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-certs" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:22.792455 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.792436 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.792571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.792476 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.792571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.792491 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.792571 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.792505 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.792730 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.792566 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-credential-socket" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:22.793648 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.793630 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-envoy" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:22.793867 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.793819 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-token" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:22.793958 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.793942 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-podinfo" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:22.794266 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.794246 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-kube-api-access-wfjzs" pod "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" (UID: "2c8b2d25-b0ad-4591-b480-c02c4e0fe43d"). InnerVolumeSpecName "kube-api-access-wfjzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:22.893397 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.893377 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.893397 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.893398 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.893549 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.893408 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.893549 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.893417 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:22.893549 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:22.893426 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfjzs\" (UniqueName: \"kubernetes.io/projected/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d-kube-api-access-wfjzs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:23.301041 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.301006 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" path="/var/lib/kubelet/pods/3c5a92fb-09f8-4caf-8a18-652b94bfa460/volumes" Oct 31 12:59:23.301398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.301384 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" path="/var/lib/kubelet/pods/4fdb1c5f-7bad-47ae-a852-e892a74f5cfa/volumes" Oct 31 12:59:23.303161 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.303134 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:23.356103 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.356081 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:59:23.462797 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.462766 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:23.597196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.597167 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" Oct 31 12:59:23.597664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.597162 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8" event={"ID":"2c8b2d25-b0ad-4591-b480-c02c4e0fe43d","Type":"ContainerDied","Data":"9b67a99e878f58621212a8f1b37dc68ae3021b244339f80659cc14862b5e668e"} Oct 31 12:59:23.597664 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.597302 6727 scope.go:122] "RemoveContainer" containerID="cd81376acc6d238b83d584633c09a56a6e7429815f9dec7b6e4e164eab3da88f" Oct 31 12:59:23.625839 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.625810 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:23.630878 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.630839 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8"] Oct 31 12:59:23.634398 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:23.634362 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d176a5-openshift-default-f64bdc855-zhrs8"] Oct 31 12:59:24.062554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:24.059848 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:24.277750 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:24.277712 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:25.057870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:25.057837 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:25.300434 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:25.300405 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" path="/var/lib/kubelet/pods/2c8b2d25-b0ad-4591-b480-c02c4e0fe43d/volumes" Oct 31 12:59:25.740829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:25.740804 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:26.439597 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:26.439565 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:26.896691 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:26.896657 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" podUID="2e47b7b7-859b-4806-8829-b41130e55cb3" containerName="payload-pre-processing" containerID="cri-o://f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60" gracePeriod=25 Oct 31 12:59:27.291843 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:27.291766 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.008616 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:28.008583 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.317551 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:28.317456 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.492163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:28.492132 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.673905 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:28.673873 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.838831 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:28.838802 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.981119 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:28.980997 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:29.774024 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:29.773965 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:30.341008 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:30.340975 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 12:59:30.584367 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:30.584334 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:31.199301 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:31.199260 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:31.349023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:31.348995 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:31.510189 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:31.510101 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:31.644256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:31.644224 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:32.007018 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:32.006989 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:32.510976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:32.510945 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:33.236804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:33.236754 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:33.934256 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:33.934224 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:34.514773 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:34.514745 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:34.657001 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:34.656965 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:34.845965 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:34.845936 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:35.200892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:35.200810 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:35.540785 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:35.540711 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:36.376109 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:36.376077 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:36.497339 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:36.497301 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" podUID="2e47b7b7-859b-4806-8829-b41130e55cb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:37.016431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:37.016399 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj"] Oct 31 12:59:37.196760 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:37.196721 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:37.467242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:37.467209 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:38.136743 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:38.136701 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:38.833656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:38.833615 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:39.518895 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:39.518859 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:39.590506 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:39.590456 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd"] Oct 31 12:59:39.598352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:39.598323 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk"] Oct 31 12:59:39.718928 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:39.718900 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:39.852351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:39.852322 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:40.241763 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:40.241672 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:40.507873 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:40.507800 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47"] Oct 31 12:59:40.508023 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:40.508000 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" containerID="cri-o://bcad8df32898d0d5792ee11c8b957d77f2da3a6975508768bc92f106b5f4402b" gracePeriod=30 Oct 31 12:59:41.042198 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:41.042168 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:41.797093 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:41.797059 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:42.082047 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:42.082011 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:42.268057 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:42.268021 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:42.459753 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:42.459685 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:42.687837 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:42.687804 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:43.548680 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:43.548644 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:44.319628 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.319594 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:44.590825 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.590738 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" podUID="48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" containerName="payload-processing" containerID="cri-o://fb2bb934a78206ea1ef26f639c24c75efd434cfe3cad51875ac35ac3dc47bde9" gracePeriod=25 Oct 31 12:59:44.599207 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.599179 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" podUID="badf7b01-8d02-47e5-877f-addd475b2921" containerName="payload-pre-processing" containerID="cri-o://832074edc2bcdd78b1d7e611e03b440ccc3f7496edfac700ae79cb4c5d242b01" gracePeriod=25 Oct 31 12:59:44.733352 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.733321 6727 generic.go:356] "Generic (PLEG): container finished" podID="badf7b01-8d02-47e5-877f-addd475b2921" containerID="832074edc2bcdd78b1d7e611e03b440ccc3f7496edfac700ae79cb4c5d242b01" exitCode=0 Oct 31 12:59:44.733514 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.733414 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" event={"ID":"badf7b01-8d02-47e5-877f-addd475b2921","Type":"ContainerDied","Data":"832074edc2bcdd78b1d7e611e03b440ccc3f7496edfac700ae79cb4c5d242b01"} Oct 31 12:59:44.736095 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.736072 6727 generic.go:356] "Generic (PLEG): container finished" podID="48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" containerID="fb2bb934a78206ea1ef26f639c24c75efd434cfe3cad51875ac35ac3dc47bde9" exitCode=0 Oct 31 12:59:44.736205 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.736142 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" event={"ID":"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e","Type":"ContainerDied","Data":"fb2bb934a78206ea1ef26f639c24c75efd434cfe3cad51875ac35ac3dc47bde9"} Oct 31 12:59:44.845250 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.845167 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w"] Oct 31 12:59:44.845438 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.845412 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" podUID="fda97a76-3a20-4ee9-9a41-dfdc3887df90" containerName="maas-api" containerID="cri-o://df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97" gracePeriod=30 Oct 31 12:59:44.954858 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.954832 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:59:44.963369 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.963345 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:59:44.995510 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:44.993770 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:45.106548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.093880 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-plugins-config-volume\") pod \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " Oct 31 12:59:45.106548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.094016 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf7b01-8d02-47e5-877f-addd475b2921-kube-api-access-flbgz\" (UniqueName: \"kubernetes.io/projected/badf7b01-8d02-47e5-877f-addd475b2921-kube-api-access-flbgz\") pod \"badf7b01-8d02-47e5-877f-addd475b2921\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " Oct 31 12:59:45.106548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.094107 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-kube-api-access-shrl8\" (UniqueName: \"kubernetes.io/projected/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-kube-api-access-shrl8\") pod \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\" (UID: \"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e\") " Oct 31 12:59:45.106548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.094295 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badf7b01-8d02-47e5-877f-addd475b2921-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf7b01-8d02-47e5-877f-addd475b2921-plugins-config-volume\") pod \"badf7b01-8d02-47e5-877f-addd475b2921\" (UID: \"badf7b01-8d02-47e5-877f-addd475b2921\") " Oct 31 12:59:45.106548 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.097321 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-plugins-config-volume" pod "48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" (UID: "48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:45.122500 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.117728 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badf7b01-8d02-47e5-877f-addd475b2921-plugins-config-volume" pod "badf7b01-8d02-47e5-877f-addd475b2921" (UID: "badf7b01-8d02-47e5-877f-addd475b2921"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:45.124385 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.124350 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:45.132482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.124740 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-kube-api-access-shrl8" pod "48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" (UID: "48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e"). InnerVolumeSpecName "kube-api-access-shrl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:45.132482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.126683 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf7b01-8d02-47e5-877f-addd475b2921-kube-api-access-flbgz" pod "badf7b01-8d02-47e5-877f-addd475b2921" (UID: "badf7b01-8d02-47e5-877f-addd475b2921"). InnerVolumeSpecName "kube-api-access-flbgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:45.176016 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.175991 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:59:45.195870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.195834 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf7b01-8d02-47e5-877f-addd475b2921-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.195870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.195867 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.195870 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.195881 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbgz\" (UniqueName: \"kubernetes.io/projected/badf7b01-8d02-47e5-877f-addd475b2921-kube-api-access-flbgz\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.196168 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.195894 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shrl8\" (UniqueName: \"kubernetes.io/projected/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e-kube-api-access-shrl8\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.296289 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.296261 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\") pod \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " Oct 31 12:59:45.296482 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.296373 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-metrics-certs\") pod \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " Oct 31 12:59:45.296593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.296573 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda97a76-3a20-4ee9-9a41-dfdc3887df90-kube-api-access-vxqjd\" (UniqueName: \"kubernetes.io/projected/fda97a76-3a20-4ee9-9a41-dfdc3887df90-kube-api-access-vxqjd\") pod \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " Oct 31 12:59:45.296675 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.296629 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda97a76-3a20-4ee9-9a41-dfdc3887df90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda97a76-3a20-4ee9-9a41-dfdc3887df90-openshift-service-ca\") pod \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\" (UID: \"fda97a76-3a20-4ee9-9a41-dfdc3887df90\") " Oct 31 12:59:45.297040 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.297014 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda97a76-3a20-4ee9-9a41-dfdc3887df90-openshift-service-ca" pod "fda97a76-3a20-4ee9-9a41-dfdc3887df90" (UID: "fda97a76-3a20-4ee9-9a41-dfdc3887df90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:45.298722 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.298693 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-metrics-certs" pod "fda97a76-3a20-4ee9-9a41-dfdc3887df90" (UID: "fda97a76-3a20-4ee9-9a41-dfdc3887df90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:45.298911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.298889 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda97a76-3a20-4ee9-9a41-dfdc3887df90-kube-api-access-vxqjd" pod "fda97a76-3a20-4ee9-9a41-dfdc3887df90" (UID: "fda97a76-3a20-4ee9-9a41-dfdc3887df90"). InnerVolumeSpecName "kube-api-access-vxqjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:45.299059 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.299037 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls" pod "fda97a76-3a20-4ee9-9a41-dfdc3887df90" (UID: "fda97a76-3a20-4ee9-9a41-dfdc3887df90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:45.300579 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.300552 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:45.398531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.398440 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqjd\" (UniqueName: \"kubernetes.io/projected/fda97a76-3a20-4ee9-9a41-dfdc3887df90-kube-api-access-vxqjd\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.398531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.398490 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda97a76-3a20-4ee9-9a41-dfdc3887df90-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.398531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.398507 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.398531 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.398520 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda97a76-3a20-4ee9-9a41-dfdc3887df90-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.434796 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.434766 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:45.744689 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.744660 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" Oct 31 12:59:45.745105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.744693 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd" event={"ID":"48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e","Type":"ContainerDied","Data":"15d703bf04dceb30c009366ff42881152b904e4b18bdd0fd4be1bfb3050c881b"} Oct 31 12:59:45.745105 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.744755 6727 scope.go:122] "RemoveContainer" containerID="fb2bb934a78206ea1ef26f639c24c75efd434cfe3cad51875ac35ac3dc47bde9" Oct 31 12:59:45.747542 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.747512 6727 generic.go:356] "Generic (PLEG): container finished" podID="fda97a76-3a20-4ee9-9a41-dfdc3887df90" containerID="df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97" exitCode=0 Oct 31 12:59:45.747666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.747566 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" Oct 31 12:59:45.747666 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.747597 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" event={"ID":"fda97a76-3a20-4ee9-9a41-dfdc3887df90","Type":"ContainerDied","Data":"df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97"} Oct 31 12:59:45.747788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.747660 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w" event={"ID":"fda97a76-3a20-4ee9-9a41-dfdc3887df90","Type":"ContainerDied","Data":"cc2375a2dba19d426eae8e9c3f48c28caca1494af2e1d018257773a8a293ecc3"} Oct 31 12:59:45.749860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.749734 6727 generic.go:356] "Generic (PLEG): container finished" podID="441559e3-5ae0-40bb-b7cc-604837dea088" containerID="bcad8df32898d0d5792ee11c8b957d77f2da3a6975508768bc92f106b5f4402b" exitCode=0 Oct 31 12:59:45.749860 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.749795 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" event={"ID":"441559e3-5ae0-40bb-b7cc-604837dea088","Type":"ContainerDied","Data":"bcad8df32898d0d5792ee11c8b957d77f2da3a6975508768bc92f106b5f4402b"} Oct 31 12:59:45.751264 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.751246 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" event={"ID":"badf7b01-8d02-47e5-877f-addd475b2921","Type":"ContainerDied","Data":"545bdd0181141dd314d20cd373e292d96cb7fbaa847cba1d37fe3a8bd46b7163"} Oct 31 12:59:45.751361 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.751254 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk" Oct 31 12:59:45.774089 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.774048 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:45.777141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.777124 6727 scope.go:122] "RemoveContainer" containerID="df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97" Oct 31 12:59:45.782127 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.782104 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:59:45.787600 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.787571 6727 scope.go:122] "RemoveContainer" containerID="df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97" Oct 31 12:59:45.787863 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:45.787845 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97\": container with ID starting with df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97 not found: ID does not exist" containerID="df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97" Oct 31 12:59:45.787912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.787873 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97"} err="failed to get container status \"df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97\": rpc error: code = NotFound desc = could not find container \"df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97\": container with ID starting with df04318d9af3be835b638fd3a61346c4d785f98d842caf44140ed7ef09910c97 not found: ID does not exist" Oct 31 12:59:45.787912 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.787890 6727 scope.go:122] "RemoveContainer" containerID="832074edc2bcdd78b1d7e611e03b440ccc3f7496edfac700ae79cb4c5d242b01" Oct 31 12:59:45.793832 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.793811 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w"] Oct 31 12:59:45.798933 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.798915 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-62734c-68fcfcfb99-p9j6w"] Oct 31 12:59:45.804293 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804275 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-data\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804401 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804312 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441559e3-5ae0-40bb-b7cc-604837dea088-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441559e3-5ae0-40bb-b7cc-604837dea088-istiod-ca-cert\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804511 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804495 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-certs\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804531 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-socket\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804539 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-data" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:45.804685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804609 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-credential-socket\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804685 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804669 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/441559e3-5ae0-40bb-b7cc-604837dea088-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441559e3-5ae0-40bb-b7cc-604837dea088-istio-podinfo\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804756 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-envoy\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804794 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804785 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441559e3-5ae0-40bb-b7cc-604837dea088-istiod-ca-cert" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:45.804901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804786 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-socket" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:45.804901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804792 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-kube-api-access-2c8qd\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-kube-api-access-2c8qd\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.804901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804809 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-credential-socket" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:45.804901 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804847 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-istio-token\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-istio-token\") pod \"441559e3-5ae0-40bb-b7cc-604837dea088\" (UID: \"441559e3-5ae0-40bb-b7cc-604837dea088\") " Oct 31 12:59:45.805096 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.804962 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-certs" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:45.805324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.805303 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.805324 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.805323 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441559e3-5ae0-40bb-b7cc-604837dea088-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.805497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.805333 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.805497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.805343 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.805497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.805352 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.806701 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.806679 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/441559e3-5ae0-40bb-b7cc-604837dea088-istio-podinfo" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:45.806872 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.806857 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-istio-token" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:45.807141 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.807122 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-envoy" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:45.807225 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.807141 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-kube-api-access-2c8qd" pod "441559e3-5ae0-40bb-b7cc-604837dea088" (UID: "441559e3-5ae0-40bb-b7cc-604837dea088"). InnerVolumeSpecName "kube-api-access-2c8qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:45.813882 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.813863 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd"] Oct 31 12:59:45.819923 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.819898 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-62734c-7886fdccdc-jvzmd"] Oct 31 12:59:45.832431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.832412 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk"] Oct 31 12:59:45.837800 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.837774 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-62734c-864bd89bf5-msbrk"] Oct 31 12:59:45.906377 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.906341 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441559e3-5ae0-40bb-b7cc-604837dea088-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.906377 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.906378 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441559e3-5ae0-40bb-b7cc-604837dea088-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.906563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.906394 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8qd\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-kube-api-access-2c8qd\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.906563 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:45.906422 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441559e3-5ae0-40bb-b7cc-604837dea088-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:46.241449 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.241408 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:46.497390 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.497309 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" podUID="2e47b7b7-859b-4806-8829-b41130e55cb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:46.761667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.761587 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" event={"ID":"441559e3-5ae0-40bb-b7cc-604837dea088","Type":"ContainerDied","Data":"0c4f1050cbd7091520eceff49c88f3cddca34d0dbb49693208fd0584046c6d18"} Oct 31 12:59:46.761667 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.761648 6727 scope.go:122] "RemoveContainer" containerID="bcad8df32898d0d5792ee11c8b957d77f2da3a6975508768bc92f106b5f4402b" Oct 31 12:59:46.762139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.761597 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47" Oct 31 12:59:46.786694 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.786671 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47"] Oct 31 12:59:46.790431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.790408 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-qxz47"] Oct 31 12:59:46.968656 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:46.968627 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:47.167527 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.167504 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:59:47.220616 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.220591 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e47b7b7-859b-4806-8829-b41130e55cb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e47b7b7-859b-4806-8829-b41130e55cb3-plugins-config-volume\") pod \"2e47b7b7-859b-4806-8829-b41130e55cb3\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " Oct 31 12:59:47.220788 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.220695 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e47b7b7-859b-4806-8829-b41130e55cb3-kube-api-access-wvfgf\" (UniqueName: \"kubernetes.io/projected/2e47b7b7-859b-4806-8829-b41130e55cb3-kube-api-access-wvfgf\") pod \"2e47b7b7-859b-4806-8829-b41130e55cb3\" (UID: \"2e47b7b7-859b-4806-8829-b41130e55cb3\") " Oct 31 12:59:47.220955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.220931 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e47b7b7-859b-4806-8829-b41130e55cb3-plugins-config-volume" pod "2e47b7b7-859b-4806-8829-b41130e55cb3" (UID: "2e47b7b7-859b-4806-8829-b41130e55cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:47.221182 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.221163 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e47b7b7-859b-4806-8829-b41130e55cb3-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:47.222893 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.222866 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e47b7b7-859b-4806-8829-b41130e55cb3-kube-api-access-wvfgf" pod "2e47b7b7-859b-4806-8829-b41130e55cb3" (UID: "2e47b7b7-859b-4806-8829-b41130e55cb3"). InnerVolumeSpecName "kube-api-access-wvfgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:47.300713 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.300655 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" path="/var/lib/kubelet/pods/441559e3-5ae0-40bb-b7cc-604837dea088/volumes" Oct 31 12:59:47.301144 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.301125 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" path="/var/lib/kubelet/pods/48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e/volumes" Oct 31 12:59:47.301578 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.301561 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badf7b01-8d02-47e5-877f-addd475b2921" path="/var/lib/kubelet/pods/badf7b01-8d02-47e5-877f-addd475b2921/volumes" Oct 31 12:59:47.302400 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.302381 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda97a76-3a20-4ee9-9a41-dfdc3887df90" path="/var/lib/kubelet/pods/fda97a76-3a20-4ee9-9a41-dfdc3887df90/volumes" Oct 31 12:59:47.322079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.322034 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvfgf\" (UniqueName: \"kubernetes.io/projected/2e47b7b7-859b-4806-8829-b41130e55cb3-kube-api-access-wvfgf\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:47.771553 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.771519 6727 generic.go:356] "Generic (PLEG): container finished" podID="2e47b7b7-859b-4806-8829-b41130e55cb3" containerID="f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60" exitCode=0 Oct 31 12:59:47.771983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.771566 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" event={"ID":"2e47b7b7-859b-4806-8829-b41130e55cb3","Type":"ContainerDied","Data":"f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60"} Oct 31 12:59:47.771983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.771568 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" Oct 31 12:59:47.771983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.771605 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6" event={"ID":"2e47b7b7-859b-4806-8829-b41130e55cb3","Type":"ContainerDied","Data":"9c4069180bc11825ff8f4f071ff4534b060635a3d4ccb145e5e5673ae96cf289"} Oct 31 12:59:47.771983 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.771626 6727 scope.go:122] "RemoveContainer" containerID="f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60" Oct 31 12:59:47.781235 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.781218 6727 scope.go:122] "RemoveContainer" containerID="f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60" Oct 31 12:59:47.781515 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:47.781491 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60\": container with ID starting with f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60 not found: ID does not exist" containerID="f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60" Oct 31 12:59:47.781604 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.781526 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60"} err="failed to get container status \"f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60\": rpc error: code = NotFound desc = could not find container \"f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60\": container with ID starting with f871a3ac4bef5ddb4682ed9fde8f768ab11708578c5ac73ad23c9126776b3b60 not found: ID does not exist" Oct 31 12:59:47.791668 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.791646 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6"] Oct 31 12:59:47.795999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.795981 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71bc05-7d54fc6ff-cxtm6"] Oct 31 12:59:47.861606 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:47.861570 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:48.528530 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:48.528494 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:48.912294 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:48.912252 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:49.047874 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:49.047836 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:49.300792 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:49.300724 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e47b7b7-859b-4806-8829-b41130e55cb3" path="/var/lib/kubelet/pods/2e47b7b7-859b-4806-8829-b41130e55cb3/volumes" Oct 31 12:59:49.328871 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:49.328838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:49.951378 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:49.951299 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 12:59:50.087490 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:50.086969 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:50.965918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:50.965874 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:51.728911 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:51.728874 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:52.036428 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.036351 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" containerID="cri-o://e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163" gracePeriod=45 Oct 31 12:59:52.301102 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.301035 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:59:52.386986 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.386956 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07021190-faf9-432c-b168-573737c7a3c4-kube-api-access-z27fg\" (UniqueName: \"kubernetes.io/projected/07021190-faf9-432c-b168-573737c7a3c4-kube-api-access-z27fg\") pod \"07021190-faf9-432c-b168-573737c7a3c4\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " Oct 31 12:59:52.387163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.387066 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-tmp-dir\") pod \"07021190-faf9-432c-b168-573737c7a3c4\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " Oct 31 12:59:52.387163 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.387153 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07021190-faf9-432c-b168-573737c7a3c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/07021190-faf9-432c-b168-573737c7a3c4-tls-certs\") pod \"07021190-faf9-432c-b168-573737c7a3c4\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " Oct 31 12:59:52.387290 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.387262 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-home\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-home\") pod \"07021190-faf9-432c-b168-573737c7a3c4\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " Oct 31 12:59:52.387351 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.387312 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-dshm\") pod \"07021190-faf9-432c-b168-573737c7a3c4\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " Oct 31 12:59:52.387399 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.387353 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-model-cache\") pod \"07021190-faf9-432c-b168-573737c7a3c4\" (UID: \"07021190-faf9-432c-b168-573737c7a3c4\") " Oct 31 12:59:52.387918 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.387858 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-model-cache" pod "07021190-faf9-432c-b168-573737c7a3c4" (UID: "07021190-faf9-432c-b168-573737c7a3c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:52.389588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.388408 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:52.389588 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.389109 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-tmp-dir" pod "07021190-faf9-432c-b168-573737c7a3c4" (UID: "07021190-faf9-432c-b168-573737c7a3c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:52.389863 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.389617 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-home" pod "07021190-faf9-432c-b168-573737c7a3c4" (UID: "07021190-faf9-432c-b168-573737c7a3c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:52.392754 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.392709 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07021190-faf9-432c-b168-573737c7a3c4-tls-certs" pod "07021190-faf9-432c-b168-573737c7a3c4" (UID: "07021190-faf9-432c-b168-573737c7a3c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:52.392865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.392798 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-dshm" pod "07021190-faf9-432c-b168-573737c7a3c4" (UID: "07021190-faf9-432c-b168-573737c7a3c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:52.392865 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.392858 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07021190-faf9-432c-b168-573737c7a3c4-kube-api-access-z27fg" pod "07021190-faf9-432c-b168-573737c7a3c4" (UID: "07021190-faf9-432c-b168-573737c7a3c4"). InnerVolumeSpecName "kube-api-access-z27fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:52.489920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.489888 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z27fg\" (UniqueName: \"kubernetes.io/projected/07021190-faf9-432c-b168-573737c7a3c4-kube-api-access-z27fg\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:52.489920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.489918 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:52.489920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.489927 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07021190-faf9-432c-b168-573737c7a3c4-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:52.490178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.489935 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:52.490178 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.489943 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07021190-faf9-432c-b168-573737c7a3c4-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:52.650246 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.650216 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:52.805756 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.805724 6727 generic.go:356] "Generic (PLEG): container finished" podID="07021190-faf9-432c-b168-573737c7a3c4" containerID="e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163" exitCode=0 Oct 31 12:59:52.805955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.805772 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" Oct 31 12:59:52.805955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.805808 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" event={"ID":"07021190-faf9-432c-b168-573737c7a3c4","Type":"ContainerDied","Data":"e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163"} Oct 31 12:59:52.805955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.805850 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj" event={"ID":"07021190-faf9-432c-b168-573737c7a3c4","Type":"ContainerDied","Data":"93f5aca03c7036221894f9a4ec967ce634307c1591662efdc9d5b4e053a42914"} Oct 31 12:59:52.805955 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.805866 6727 scope.go:122] "RemoveContainer" containerID="e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163" Oct 31 12:59:52.816079 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.816062 6727 scope.go:122] "RemoveContainer" containerID="e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163" Oct 31 12:59:52.816327 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:52.816305 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163\": container with ID starting with e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163 not found: ID does not exist" containerID="e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163" Oct 31 12:59:52.816384 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.816335 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163"} err="failed to get container status \"e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163\": rpc error: code = NotFound desc = could not find container \"e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163\": container with ID starting with e99b56d21e07c229b26ac7580b98bcb0247858166ba83bf3c0ab595f41850163 not found: ID does not exist" Oct 31 12:59:52.828609 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.828585 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj"] Oct 31 12:59:52.833055 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:52.833035 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/premium-w5-b25d3b-kserve-849bd65849-xzdpj"] Oct 31 12:59:53.043496 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.043403 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9"] Oct 31 12:59:53.070593 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.070558 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv"] Oct 31 12:59:53.195982 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.195946 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:53.305985 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.305894 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07021190-faf9-432c-b168-573737c7a3c4" path="/var/lib/kubelet/pods/07021190-faf9-432c-b168-573737c7a3c4/volumes" Oct 31 12:59:53.342829 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.342789 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:53.479026 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.477726 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:53.661106 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.661065 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:53.807395 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.807364 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:53.981917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:53.981795 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:54.308392 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:54.308116 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:55.262192 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:55.262158 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:55.863067 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:55.863033 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq"] Oct 31 12:59:55.863497 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:55.863222 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" podUID="e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" containerName="maas-api" containerID="cri-o://a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020" gracePeriod=30 Oct 31 12:59:55.938139 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:55.938115 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:56.156135 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.156104 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:56.171826 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.171804 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:59:56.242890 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.242862 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-openshift-service-ca\") pod \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " Oct 31 12:59:56.243031 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.242969 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-kube-api-access-l5bvl\" (UniqueName: \"kubernetes.io/projected/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-kube-api-access-l5bvl\") pod \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " Oct 31 12:59:56.243157 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.243136 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-openshift-service-ca" pod "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" (UID: "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:56.243196 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.243139 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-maas-api-tls\") pod \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " Oct 31 12:59:56.243232 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.243198 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-metrics-certs\") pod \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\" (UID: \"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6\") " Oct 31 12:59:56.243633 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.243616 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.245166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.245135 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-metrics-certs" pod "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" (UID: "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:56.245166 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.245149 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-kube-api-access-l5bvl" pod "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" (UID: "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6"). InnerVolumeSpecName "kube-api-access-l5bvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:56.245271 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.245202 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-maas-api-tls" pod "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" (UID: "e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:56.300602 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.300578 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:56.345061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.345027 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.345061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.345058 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.345061 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.345069 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5bvl\" (UniqueName: \"kubernetes.io/projected/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6-kube-api-access-l5bvl\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.439544 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.439445 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:56.730564 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.730492 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:56.832935 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.832897 6727 generic.go:356] "Generic (PLEG): container finished" podID="e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" containerID="a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020" exitCode=0 Oct 31 12:59:56.833130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.832935 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" event={"ID":"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6","Type":"ContainerDied","Data":"a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020"} Oct 31 12:59:56.833130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.832977 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" Oct 31 12:59:56.833130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.833019 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq" event={"ID":"e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6","Type":"ContainerDied","Data":"ec31dbeaae01049d2a1d0584c7cdbaed2eadecd150380194f53af0125da08fde"} Oct 31 12:59:56.833130 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.833042 6727 scope.go:122] "RemoveContainer" containerID="a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020" Oct 31 12:59:56.845903 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.845888 6727 scope.go:122] "RemoveContainer" containerID="a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020" Oct 31 12:59:56.846144 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:56.846122 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020\": container with ID starting with a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020 not found: ID does not exist" containerID="a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020" Oct 31 12:59:56.846242 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.846151 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020"} err="failed to get container status \"a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020\": rpc error: code = NotFound desc = could not find container \"a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020\": container with ID starting with a40cdbf11aaa2b5a4156784141861896cbb0de766ea243b7d331926b51091020 not found: ID does not exist" Oct 31 12:59:56.859052 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.859019 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq"] Oct 31 12:59:56.864110 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:56.864081 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9b17b9-68cbc647dd-4pshq"] Oct 31 12:59:57.128507 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:57.125507 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:57.300002 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:57.299971 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" path="/var/lib/kubelet/pods/e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6/volumes" Oct 31 12:59:57.859082 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:57.859052 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:58.044724 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.044673 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" podUID="1d707124-abc0-4daa-8a91-0af9e6081951" containerName="payload-processing" containerID="cri-o://bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c" gracePeriod=25 Oct 31 12:59:58.071085 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.071050 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" podUID="f28e1040-0b3d-45c3-aa1b-80a1856a3e10" containerName="payload-pre-processing" containerID="cri-o://7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a" gracePeriod=25 Oct 31 12:59:58.339728 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.339699 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:59:58.362285 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.362264 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:59:58.468585 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.468511 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-plugins-config-volume\") pod \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " Oct 31 12:59:58.468735 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.468681 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-kube-api-access-4lr2m\" (UniqueName: \"kubernetes.io/projected/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-kube-api-access-4lr2m\") pod \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\" (UID: \"f28e1040-0b3d-45c3-aa1b-80a1856a3e10\") " Oct 31 12:59:58.468806 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.468748 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d707124-abc0-4daa-8a91-0af9e6081951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d707124-abc0-4daa-8a91-0af9e6081951-plugins-config-volume\") pod \"1d707124-abc0-4daa-8a91-0af9e6081951\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " Oct 31 12:59:58.468896 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.468883 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d707124-abc0-4daa-8a91-0af9e6081951-kube-api-access-hklsn\" (UniqueName: \"kubernetes.io/projected/1d707124-abc0-4daa-8a91-0af9e6081951-kube-api-access-hklsn\") pod \"1d707124-abc0-4daa-8a91-0af9e6081951\" (UID: \"1d707124-abc0-4daa-8a91-0af9e6081951\") " Oct 31 12:59:58.468976 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.468951 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-plugins-config-volume" pod "f28e1040-0b3d-45c3-aa1b-80a1856a3e10" (UID: "f28e1040-0b3d-45c3-aa1b-80a1856a3e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:58.469094 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.469070 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d707124-abc0-4daa-8a91-0af9e6081951-plugins-config-volume" pod "1d707124-abc0-4daa-8a91-0af9e6081951" (UID: "1d707124-abc0-4daa-8a91-0af9e6081951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:58.469366 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.469343 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:58.469431 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.469373 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d707124-abc0-4daa-8a91-0af9e6081951-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:58.470907 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.470886 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-kube-api-access-4lr2m" pod "f28e1040-0b3d-45c3-aa1b-80a1856a3e10" (UID: "f28e1040-0b3d-45c3-aa1b-80a1856a3e10"). InnerVolumeSpecName "kube-api-access-4lr2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:58.471026 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.471010 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d707124-abc0-4daa-8a91-0af9e6081951-kube-api-access-hklsn" pod "1d707124-abc0-4daa-8a91-0af9e6081951" (UID: "1d707124-abc0-4daa-8a91-0af9e6081951"). InnerVolumeSpecName "kube-api-access-hklsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:58.543960 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.543928 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:58.570692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.570660 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hklsn\" (UniqueName: \"kubernetes.io/projected/1d707124-abc0-4daa-8a91-0af9e6081951-kube-api-access-hklsn\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:58.570692 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.570686 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lr2m\" (UniqueName: \"kubernetes.io/projected/f28e1040-0b3d-45c3-aa1b-80a1856a3e10-kube-api-access-4lr2m\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:58.847545 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.847517 6727 generic.go:356] "Generic (PLEG): container finished" podID="f28e1040-0b3d-45c3-aa1b-80a1856a3e10" containerID="7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a" exitCode=0 Oct 31 12:59:58.847723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.847561 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" Oct 31 12:59:58.847723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.847598 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" event={"ID":"f28e1040-0b3d-45c3-aa1b-80a1856a3e10","Type":"ContainerDied","Data":"7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a"} Oct 31 12:59:58.847723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.847648 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv" event={"ID":"f28e1040-0b3d-45c3-aa1b-80a1856a3e10","Type":"ContainerDied","Data":"af93daeaaae5f1120503842e1fdb15437dad621cf3c6c6ba7a6982328ff7ef9f"} Oct 31 12:59:58.847723 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.847676 6727 scope.go:122] "RemoveContainer" containerID="7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a" Oct 31 12:59:58.849169 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.849146 6727 generic.go:356] "Generic (PLEG): container finished" podID="1d707124-abc0-4daa-8a91-0af9e6081951" containerID="bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c" exitCode=0 Oct 31 12:59:58.849302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.849195 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" Oct 31 12:59:58.849302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.849222 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" event={"ID":"1d707124-abc0-4daa-8a91-0af9e6081951","Type":"ContainerDied","Data":"bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c"} Oct 31 12:59:58.849302 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.849274 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9" event={"ID":"1d707124-abc0-4daa-8a91-0af9e6081951","Type":"ContainerDied","Data":"59135057d2d629e3ae840362588bb56dda061190c88b21586e34a8635c4edc55"} Oct 31 12:59:58.857999 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.857982 6727 scope.go:122] "RemoveContainer" containerID="7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a" Oct 31 12:59:58.858310 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:58.858287 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a\": container with ID starting with 7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a not found: ID does not exist" containerID="7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a" Oct 31 12:59:58.858424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.858317 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a"} err="failed to get container status \"7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a\": rpc error: code = NotFound desc = could not find container \"7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a\": container with ID starting with 7058446bc34b497fee68730e8af99102ae2f4d117d14d66265206cb45bc25e8a not found: ID does not exist" Oct 31 12:59:58.858424 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.858340 6727 scope.go:122] "RemoveContainer" containerID="bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c" Oct 31 12:59:58.867412 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.867398 6727 scope.go:122] "RemoveContainer" containerID="bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c" Oct 31 12:59:58.867654 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:58.867635 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c\": container with ID starting with bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c not found: ID does not exist" containerID="bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c" Oct 31 12:59:58.867716 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.867659 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c"} err="failed to get container status \"bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c\": rpc error: code = NotFound desc = could not find container \"bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c\": container with ID starting with bf81bed60cd463ac05d8cb307f225db3a7a09bf14e8b20c37f28eb983352178c not found: ID does not exist" Oct 31 12:59:58.873700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.873668 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9"] Oct 31 12:59:58.880193 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.880172 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9b17b9-687cf77b4f-f89d9"] Oct 31 12:59:58.892804 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.892786 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv"] Oct 31 12:59:58.899539 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:58.899517 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9b17b9-79579fc7dc-vxvhv"] Oct 31 12:59:59.062353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.062146 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6"] Oct 31 12:59:59.062353 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.062335 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" podUID="f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" containerName="maas-api" containerID="cri-o://39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c" gracePeriod=30 Oct 31 12:59:59.142824 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.142761 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.321015 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.320966 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d707124-abc0-4daa-8a91-0af9e6081951" path="/var/lib/kubelet/pods/1d707124-abc0-4daa-8a91-0af9e6081951/volumes" Oct 31 12:59:59.321747 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.321708 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28e1040-0b3d-45c3-aa1b-80a1856a3e10" path="/var/lib/kubelet/pods/f28e1040-0b3d-45c3-aa1b-80a1856a3e10/volumes" Oct 31 12:59:59.322525 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.322496 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.349410 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.349382 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:59:59.479673 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.479570 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.486280 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.486253 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-openshift-service-ca\") pod \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " Oct 31 12:59:59.486406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.486346 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-maas-api-tls\") pod \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " Oct 31 12:59:59.486406 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.486379 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-kube-api-access-x8z4w\") pod \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " Oct 31 12:59:59.486554 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.486439 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-metrics-certs\") pod \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\" (UID: \"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0\") " Oct 31 12:59:59.486700 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.486670 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-openshift-service-ca" pod "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" (UID: "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:59.487073 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.487051 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:59.488651 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.488619 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-maas-api-tls" pod "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" (UID: "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:59.488934 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.488913 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-metrics-certs" pod "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" (UID: "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:59.489064 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.489042 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-kube-api-access-x8z4w" pod "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" (UID: "f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0"). InnerVolumeSpecName "kube-api-access-x8z4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:59.592502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.589200 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:59.592502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.589276 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-kube-api-access-x8z4w\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:59.592502 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.589294 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:59.664247 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.664218 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.856772 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.856743 6727 generic.go:356] "Generic (PLEG): container finished" podID="f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" containerID="39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c" exitCode=0 Oct 31 12:59:59.856917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.856794 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" Oct 31 12:59:59.856917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.856822 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" event={"ID":"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0","Type":"ContainerDied","Data":"39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c"} Oct 31 12:59:59.856917 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.856904 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6" event={"ID":"f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0","Type":"ContainerDied","Data":"b00ff065afbed39a3917b70bc60a83dc557c441adc7c3756a07a0cc04f92e7b9"} Oct 31 12:59:59.857088 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.856934 6727 scope.go:122] "RemoveContainer" containerID="39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c" Oct 31 12:59:59.867300 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.867279 6727 scope.go:122] "RemoveContainer" containerID="39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c" Oct 31 12:59:59.867606 ip-10-0-142-45 kubenswrapper[6727]: E1031 12:59:59.867586 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c\": container with ID starting with 39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c not found: ID does not exist" containerID="39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c" Oct 31 12:59:59.867892 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.867872 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c"} err="failed to get container status \"39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c\": rpc error: code = NotFound desc = could not find container \"39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c\": container with ID starting with 39e9f192b721dba349f40554fcbb3536b8417d02a3bf82dc9d28e7e04fa2955c not found: ID does not exist" Oct 31 12:59:59.885515 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.885491 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6"] Oct 31 12:59:59.895920 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.895898 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b25d3b-5c468f6f88-hcpw6"] Oct 31 12:59:59.966669 ip-10-0-142-45 kubenswrapper[6727]: I1031 12:59:59.966644 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:00.339291 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:00.339232 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:00.346268 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:00.346219 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:00:01.049938 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:01.049889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:01.301425 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:01.301352 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" path="/var/lib/kubelet/pods/f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0/volumes" Oct 31 13:00:01.748898 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:01.748860 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:02.545669 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:02.545633 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:03.313911 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:03.313886 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.054227 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.054195 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.181316 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.181284 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:00:04.202162 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.202136 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.428690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.428660 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.628899 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.628862 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll"] Oct 31 13:00:04.728704 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.728629 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.804785 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.804751 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp"] Oct 31 13:00:04.805002 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.804975 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" podUID="ab740d09-6da9-4783-b693-8e6d88b8b5ee" containerName="maas-api" containerID="cri-o://32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012" gracePeriod=30 Oct 31 13:00:04.904045 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:04.904015 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:05.076490 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.076440 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 13:00:05.107760 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.107724 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:05.166670 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.166642 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-metrics-certs\") pod \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " Oct 31 13:00:05.166670 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.166673 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-maas-api-tls\") pod \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " Oct 31 13:00:05.166873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.166780 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab740d09-6da9-4783-b693-8e6d88b8b5ee-kube-api-access-kc8gw\" (UniqueName: \"kubernetes.io/projected/ab740d09-6da9-4783-b693-8e6d88b8b5ee-kube-api-access-kc8gw\") pod \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " Oct 31 13:00:05.166934 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.166875 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab740d09-6da9-4783-b693-8e6d88b8b5ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab740d09-6da9-4783-b693-8e6d88b8b5ee-openshift-service-ca\") pod \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\" (UID: \"ab740d09-6da9-4783-b693-8e6d88b8b5ee\") " Oct 31 13:00:05.167236 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.167202 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab740d09-6da9-4783-b693-8e6d88b8b5ee-openshift-service-ca" pod "ab740d09-6da9-4783-b693-8e6d88b8b5ee" (UID: "ab740d09-6da9-4783-b693-8e6d88b8b5ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:05.167406 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.167384 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab740d09-6da9-4783-b693-8e6d88b8b5ee-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:05.168901 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.168873 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-maas-api-tls" pod "ab740d09-6da9-4783-b693-8e6d88b8b5ee" (UID: "ab740d09-6da9-4783-b693-8e6d88b8b5ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:05.169355 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.169333 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-metrics-certs" pod "ab740d09-6da9-4783-b693-8e6d88b8b5ee" (UID: "ab740d09-6da9-4783-b693-8e6d88b8b5ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:05.169355 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.169344 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab740d09-6da9-4783-b693-8e6d88b8b5ee-kube-api-access-kc8gw" pod "ab740d09-6da9-4783-b693-8e6d88b8b5ee" (UID: "ab740d09-6da9-4783-b693-8e6d88b8b5ee"). InnerVolumeSpecName "kube-api-access-kc8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:05.269046 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.269017 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8gw\" (UniqueName: \"kubernetes.io/projected/ab740d09-6da9-4783-b693-8e6d88b8b5ee-kube-api-access-kc8gw\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:05.269046 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.269044 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:05.269324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.269055 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab740d09-6da9-4783-b693-8e6d88b8b5ee-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:05.344906 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.344869 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:05.907363 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.907327 6727 generic.go:356] "Generic (PLEG): container finished" podID="ab740d09-6da9-4783-b693-8e6d88b8b5ee" containerID="32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012" exitCode=0 Oct 31 13:00:05.907840 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.907372 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" Oct 31 13:00:05.907840 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.907407 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" event={"ID":"ab740d09-6da9-4783-b693-8e6d88b8b5ee","Type":"ContainerDied","Data":"32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012"} Oct 31 13:00:05.907840 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.907487 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp" event={"ID":"ab740d09-6da9-4783-b693-8e6d88b8b5ee","Type":"ContainerDied","Data":"542648d0caa1fb4e60034ad1dcefc4bc560a6d3041d4e407fae4012d5db5259a"} Oct 31 13:00:05.907840 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.907496 6727 scope.go:122] "RemoveContainer" containerID="32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012" Oct 31 13:00:05.917536 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.917519 6727 scope.go:122] "RemoveContainer" containerID="32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012" Oct 31 13:00:05.917776 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:05.917757 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012\": container with ID starting with 32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012 not found: ID does not exist" containerID="32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012" Oct 31 13:00:05.917820 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.917784 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012"} err="failed to get container status \"32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012\": rpc error: code = NotFound desc = could not find container \"32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012\": container with ID starting with 32cefe0eef99c17e235c6c2e81652c29283c3753eb3bc2d6c42e171e568eb012 not found: ID does not exist" Oct 31 13:00:05.925079 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.925057 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp"] Oct 31 13:00:05.927602 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:05.927581 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8faad6-87b7fb859-5f4zp"] Oct 31 13:00:06.144166 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:06.144132 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:06.939319 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:06.939253 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:07.300440 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:07.300351 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab740d09-6da9-4783-b693-8e6d88b8b5ee" path="/var/lib/kubelet/pods/ab740d09-6da9-4783-b693-8e6d88b8b5ee/volumes" Oct 31 13:00:07.380858 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:07.380831 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:07.525441 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:07.525408 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:07.662160 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:07.662131 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn"] Oct 31 13:00:07.663755 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:07.663726 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" containerID="cri-o://67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff" gracePeriod=30 Oct 31 13:00:07.710106 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:07.710076 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:07.945416 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:07.945321 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:08.541394 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:08.541357 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:08.676082 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:08.676051 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:08.820355 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:08.820310 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:08.983378 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:08.983343 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:09.145606 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.145529 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:09.279196 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.279159 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:09.542497 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.542395 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:09.629576 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.629507 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" podUID="995e8461-e024-4e86-899d-b5cf67abb1e0" containerName="payload-pre-processing" containerID="cri-o://b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273" gracePeriod=25 Oct 31 13:00:09.887243 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.887214 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 13:00:09.931999 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.931971 6727 generic.go:356] "Generic (PLEG): container finished" podID="995e8461-e024-4e86-899d-b5cf67abb1e0" containerID="b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273" exitCode=0 Oct 31 13:00:09.932176 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.932030 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" Oct 31 13:00:09.932176 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.932068 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" event={"ID":"995e8461-e024-4e86-899d-b5cf67abb1e0","Type":"ContainerDied","Data":"b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273"} Oct 31 13:00:09.932176 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.932129 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll" event={"ID":"995e8461-e024-4e86-899d-b5cf67abb1e0","Type":"ContainerDied","Data":"d6e4105c99b1d3e70e6f02a9a21cb4ec1a17a97093ac91c16be950c225348c5f"} Oct 31 13:00:09.932176 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.932160 6727 scope.go:122] "RemoveContainer" containerID="b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273" Oct 31 13:00:09.941594 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.941578 6727 scope.go:122] "RemoveContainer" containerID="b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273" Oct 31 13:00:09.941869 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:09.941850 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273\": container with ID starting with b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273 not found: ID does not exist" containerID="b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273" Oct 31 13:00:09.941943 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:09.941880 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273"} err="failed to get container status \"b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273\": rpc error: code = NotFound desc = could not find container \"b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273\": container with ID starting with b1c45e58e9a55b1ee01336f398d4349cf583e5142b0c1a6ae98207dc4cc88273 not found: ID does not exist" Oct 31 13:00:10.024166 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.024142 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995e8461-e024-4e86-899d-b5cf67abb1e0-kube-api-access-2fkrr\" (UniqueName: \"kubernetes.io/projected/995e8461-e024-4e86-899d-b5cf67abb1e0-kube-api-access-2fkrr\") pod \"995e8461-e024-4e86-899d-b5cf67abb1e0\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " Oct 31 13:00:10.024582 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.024206 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995e8461-e024-4e86-899d-b5cf67abb1e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8461-e024-4e86-899d-b5cf67abb1e0-plugins-config-volume\") pod \"995e8461-e024-4e86-899d-b5cf67abb1e0\" (UID: \"995e8461-e024-4e86-899d-b5cf67abb1e0\") " Oct 31 13:00:10.024582 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.024554 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995e8461-e024-4e86-899d-b5cf67abb1e0-plugins-config-volume" pod "995e8461-e024-4e86-899d-b5cf67abb1e0" (UID: "995e8461-e024-4e86-899d-b5cf67abb1e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:10.026168 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.026140 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995e8461-e024-4e86-899d-b5cf67abb1e0-kube-api-access-2fkrr" pod "995e8461-e024-4e86-899d-b5cf67abb1e0" (UID: "995e8461-e024-4e86-899d-b5cf67abb1e0"). InnerVolumeSpecName "kube-api-access-2fkrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:10.125281 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.125260 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fkrr\" (UniqueName: \"kubernetes.io/projected/995e8461-e024-4e86-899d-b5cf67abb1e0-kube-api-access-2fkrr\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:10.125386 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.125283 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995e8461-e024-4e86-899d-b5cf67abb1e0-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:10.251841 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.251808 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:10.259011 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.258955 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll"] Oct 31 13:00:10.269107 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.269073 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8faad6-588cfc8df5-wflll"] Oct 31 13:00:10.926426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:10.926395 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:11.300104 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:11.300026 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995e8461-e024-4e86-899d-b5cf67abb1e0" path="/var/lib/kubelet/pods/995e8461-e024-4e86-899d-b5cf67abb1e0/volumes" Oct 31 13:00:11.629239 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:11.629205 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:12.346882 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.346847 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:12.938840 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.938783 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 13:00:12.959792 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.959758 6727 generic.go:356] "Generic (PLEG): container finished" podID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerID="67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff" exitCode=0 Oct 31 13:00:12.959933 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.959909 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" event={"ID":"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6","Type":"ContainerDied","Data":"67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff"} Oct 31 13:00:12.959987 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.959947 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" event={"ID":"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6","Type":"ContainerDied","Data":"50c0e43e289e53b3f16b68e420f6dd3b95b5fc28127dd74103c393454b4be2ee"} Oct 31 13:00:12.959987 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.959968 6727 scope.go:122] "RemoveContainer" containerID="67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff" Oct 31 13:00:12.960108 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.960092 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn" Oct 31 13:00:12.961018 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.960967 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-socket\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961078 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istiod-ca-cert\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961111 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-credential-socket\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961157 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-data\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961233 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-podinfo\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961268 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-envoy\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961297 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-kube-api-access-zxp89\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-kube-api-access-zxp89\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961667 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961383 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-certs\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.961667 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.961401 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-token\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-token\") pod \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\" (UID: \"ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6\") " Oct 31 13:00:12.962439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.962175 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-socket" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:12.962439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.962385 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:12.962439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.962390 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-certs" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:12.962439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.962434 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-data" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:12.962770 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.962740 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istiod-ca-cert" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:12.963354 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.963325 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-credential-socket" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:12.965811 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.965342 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-kube-api-access-zxp89" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "kube-api-access-zxp89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:12.966393 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.966341 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-envoy" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:12.971640 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.971614 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-token" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:12.973591 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:12.973554 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-podinfo" pod "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" (UID: "ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:00:13.002307 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.002284 6727 scope.go:122] "RemoveContainer" containerID="67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff" Oct 31 13:00:13.002705 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:13.002676 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff\": container with ID starting with 67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff not found: ID does not exist" containerID="67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff" Oct 31 13:00:13.002786 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.002707 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff"} err="failed to get container status \"67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff\": rpc error: code = NotFound desc = could not find container \"67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff\": container with ID starting with 67885524404fe9a2598c52ee9bc106c673789a2586e61e07caae85ecdff770ff not found: ID does not exist" Oct 31 13:00:13.047057 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.047026 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:13.063357 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063325 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.063357 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063352 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.063357 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063364 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.063694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063377 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.063694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063391 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.063694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063403 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxp89\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-kube-api-access-zxp89\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.063694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063415 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.063694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.063427 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:13.283517 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.283454 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn"] Oct 31 13:00:13.289636 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.289609 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-pstxn"] Oct 31 13:00:13.301085 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.301060 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" path="/var/lib/kubelet/pods/ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6/volumes" Oct 31 13:00:13.643301 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.643258 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:00:13.750624 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.750579 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:13.881351 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:13.881323 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:14.006262 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:14.006181 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:14.538640 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:14.538610 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:14.962829 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:14.962795 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl"] Oct 31 13:00:15.073860 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.073824 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:15.133078 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.133038 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq"] Oct 31 13:00:15.133296 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.133271 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" podUID="7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" containerName="maas-api" containerID="cri-o://0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc" gracePeriod=30 Oct 31 13:00:15.236358 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.236293 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:15.436612 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.436585 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 13:00:15.441661 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.441637 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:15.509668 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.509602 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-kube-api-access-bf7zc\" (UniqueName: \"kubernetes.io/projected/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-kube-api-access-bf7zc\") pod \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " Oct 31 13:00:15.509668 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.509646 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-openshift-service-ca\") pod \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " Oct 31 13:00:15.509870 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.509832 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-metrics-certs\") pod \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " Oct 31 13:00:15.509924 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.509875 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls\") pod \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\" (UID: \"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67\") " Oct 31 13:00:15.510093 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.510068 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-openshift-service-ca" pod "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" (UID: "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:15.510365 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.510348 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:15.511900 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.511881 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-kube-api-access-bf7zc" pod "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" (UID: "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67"). InnerVolumeSpecName "kube-api-access-bf7zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:15.512006 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.511981 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls" pod "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" (UID: "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:15.512053 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.511997 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-metrics-certs" pod "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" (UID: "7543a97c-b2c1-46ef-82bc-57bb1ecd2e67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:15.611431 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.611405 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:15.611558 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.611433 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:15.611558 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.611447 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf7zc\" (UniqueName: \"kubernetes.io/projected/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67-kube-api-access-bf7zc\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:15.623017 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.622276 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:15.986501 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.986442 6727 generic.go:356] "Generic (PLEG): container finished" podID="7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" containerID="0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc" exitCode=0 Oct 31 13:00:15.986988 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.986513 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" Oct 31 13:00:15.986988 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.986526 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" event={"ID":"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67","Type":"ContainerDied","Data":"0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc"} Oct 31 13:00:15.986988 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.986578 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq" event={"ID":"7543a97c-b2c1-46ef-82bc-57bb1ecd2e67","Type":"ContainerDied","Data":"7382723185b31f1504d9330488614fa1266b839407a7ec2e791ce223bc908634"} Oct 31 13:00:15.986988 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.986604 6727 scope.go:122] "RemoveContainer" containerID="0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc" Oct 31 13:00:15.996703 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.996685 6727 scope.go:122] "RemoveContainer" containerID="0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc" Oct 31 13:00:15.996999 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:15.996979 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc\": container with ID starting with 0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc not found: ID does not exist" containerID="0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc" Oct 31 13:00:15.997061 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:15.997006 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc"} err="failed to get container status \"0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc\": rpc error: code = NotFound desc = could not find container \"0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc\": container with ID starting with 0b7110f6370636a0e8921b7d16c9df77294166bfa90d293e5d5b1e8d3ef972dc not found: ID does not exist" Oct 31 13:00:16.010207 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:16.010184 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq"] Oct 31 13:00:16.019308 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:16.019288 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2d6201-6d958df4c8-clhfq"] Oct 31 13:00:16.446490 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:16.446437 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:16.851692 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:16.851663 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:16.986077 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:16.986029 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j"] Oct 31 13:00:16.986244 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:16.986221 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" containerID="cri-o://304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54" gracePeriod=30 Oct 31 13:00:17.240764 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:17.240680 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:17.300070 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:17.300030 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" path="/var/lib/kubelet/pods/7543a97c-b2c1-46ef-82bc-57bb1ecd2e67/volumes" Oct 31 13:00:17.461641 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:17.461604 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:17.598231 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:17.598193 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:17.756646 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:17.756614 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:18.080677 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:18.080651 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:18.220608 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:18.220581 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:18.363863 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:18.363781 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:18.681808 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:18.681718 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:19.115527 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:19.115458 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:19.963200 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:19.963159 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" podUID="25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" containerName="payload-processing" containerID="cri-o://4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92" gracePeriod=25 Oct 31 13:00:20.147458 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.147365 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:20.229331 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.229271 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 13:00:20.265066 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.265039 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-kube-api-access-cmntt\" (UniqueName: \"kubernetes.io/projected/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-kube-api-access-cmntt\") pod \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " Oct 31 13:00:20.265252 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.265228 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-plugins-config-volume\") pod \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\" (UID: \"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21\") " Oct 31 13:00:20.265658 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.265621 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-plugins-config-volume" pod "25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" (UID: "25da6fc3-3543-4a22-bcb5-efbc8f5f1a21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:20.267656 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.267630 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-kube-api-access-cmntt" pod "25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" (UID: "25da6fc3-3543-4a22-bcb5-efbc8f5f1a21"). InnerVolumeSpecName "kube-api-access-cmntt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:20.366985 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.366952 6727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-plugins-config-volume\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:20.366985 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.366979 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmntt\" (UniqueName: \"kubernetes.io/projected/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21-kube-api-access-cmntt\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:20.925658 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:20.925625 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:21.016132 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.016099 6727 generic.go:356] "Generic (PLEG): container finished" podID="25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" containerID="4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92" exitCode=0 Oct 31 13:00:21.016664 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.016146 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" Oct 31 13:00:21.016664 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.016176 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" event={"ID":"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21","Type":"ContainerDied","Data":"4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92"} Oct 31 13:00:21.016664 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.016241 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" event={"ID":"25da6fc3-3543-4a22-bcb5-efbc8f5f1a21","Type":"ContainerDied","Data":"96b6ea8b2c170fc6957077ba763d26ed4d2eeb7b7ed17433f4d8c2b679019852"} Oct 31 13:00:21.016664 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.016273 6727 scope.go:122] "RemoveContainer" containerID="4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92" Oct 31 13:00:21.035325 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.035298 6727 scope.go:122] "RemoveContainer" containerID="4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92" Oct 31 13:00:21.035635 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:21.035614 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92\": container with ID starting with 4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92 not found: ID does not exist" containerID="4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92" Oct 31 13:00:21.035727 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.035641 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92"} err="failed to get container status \"4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92\": rpc error: code = NotFound desc = could not find container \"4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92\": container with ID starting with 4c2132ae78e4e28ee90a6599e365439cd054cc58547be57378f3763cb1d6ae92 not found: ID does not exist" Oct 31 13:00:21.045867 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.045845 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl"] Oct 31 13:00:21.050047 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.050027 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl"] Oct 31 13:00:21.193090 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.193011 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2d6201-6c9c66d45-2q7sl" podUID="25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.95:50052\" within 1s: context deadline exceeded" Oct 31 13:00:21.193219 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:00:21.193109 6727 logging.go:55] [core] [Channel #3840 SubChannel #3841]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.95:50052", ServerName: "10.134.0.95:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.95:50052: operation was canceled" Oct 31 13:00:21.300164 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.300139 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" path="/var/lib/kubelet/pods/25da6fc3-3543-4a22-bcb5-efbc8f5f1a21/volumes" Oct 31 13:00:21.637426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:21.637175 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:22.247321 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.247297 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 13:00:22.325001 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.324966 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:22.387409 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.387149 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-istio-token\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-istio-token\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.387409 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.387203 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-data\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.387409 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.387279 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-socket\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.388120 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.387833 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-envoy\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.388120 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.387835 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-socket" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:22.388120 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.387902 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67586435-3054-4099-808d-8482a5449e01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67586435-3054-4099-808d-8482a5449e01-istiod-ca-cert\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.388120 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.387969 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67586435-3054-4099-808d-8482a5449e01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67586435-3054-4099-808d-8482a5449e01-istio-podinfo\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.388418 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388128 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-kube-api-access-srxws\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-kube-api-access-srxws\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.388418 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388191 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-certs\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.388418 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388224 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-credential-socket\") pod \"67586435-3054-4099-808d-8482a5449e01\" (UID: \"67586435-3054-4099-808d-8482a5449e01\") " Oct 31 13:00:22.388607 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388538 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-data" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:22.388781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388763 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.388910 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388793 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.388910 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388867 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-credential-socket" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:22.389081 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388821 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67586435-3054-4099-808d-8482a5449e01-istiod-ca-cert" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:22.389081 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.388988 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-certs" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:22.392648 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.391352 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-istio-token" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:22.394549 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.394498 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-kube-api-access-srxws" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "kube-api-access-srxws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:22.395439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.395414 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67586435-3054-4099-808d-8482a5449e01-istio-podinfo" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:00:22.396869 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.396835 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-envoy" pod "67586435-3054-4099-808d-8482a5449e01" (UID: "67586435-3054-4099-808d-8482a5449e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:22.489578 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.489548 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srxws\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-kube-api-access-srxws\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.489578 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.489577 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.489741 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.489586 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.489741 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.489595 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67586435-3054-4099-808d-8482a5449e01-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.489741 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.489603 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67586435-3054-4099-808d-8482a5449e01-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.489741 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.489611 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67586435-3054-4099-808d-8482a5449e01-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.489741 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:22.489620 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67586435-3054-4099-808d-8482a5449e01-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:23.024816 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.024782 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:23.035312 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.035254 6727 generic.go:356] "Generic (PLEG): container finished" podID="67586435-3054-4099-808d-8482a5449e01" containerID="304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54" exitCode=0 Oct 31 13:00:23.035537 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.035341 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" event={"ID":"67586435-3054-4099-808d-8482a5449e01","Type":"ContainerDied","Data":"304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54"} Oct 31 13:00:23.035537 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.035411 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" event={"ID":"67586435-3054-4099-808d-8482a5449e01","Type":"ContainerDied","Data":"2f06b9a36c59dd6ae599cc6f2f905c7187673bc16d4d03a8388582ceb7b70218"} Oct 31 13:00:23.035537 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.035454 6727 scope.go:122] "RemoveContainer" containerID="304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54" Oct 31 13:00:23.035734 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.035558 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j" Oct 31 13:00:23.055502 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.055456 6727 scope.go:122] "RemoveContainer" containerID="304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54" Oct 31 13:00:23.055840 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:23.055813 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54\": container with ID starting with 304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54 not found: ID does not exist" containerID="304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54" Oct 31 13:00:23.055932 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.055853 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54"} err="failed to get container status \"304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54\": rpc error: code = NotFound desc = could not find container \"304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54\": container with ID starting with 304bf3d6b5f6d5f8525f65eb496f49d03a4d72c5b7a3e5cdf4f29e487f027e54 not found: ID does not exist" Oct 31 13:00:23.064100 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.064080 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j"] Oct 31 13:00:23.068909 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.068888 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v8c4j"] Oct 31 13:00:23.129511 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.129487 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh"] Oct 31 13:00:23.134528 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.134505 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv"] Oct 31 13:00:23.143996 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.143969 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz"] Oct 31 13:00:23.291572 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.291493 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:23.300025 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.299989 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67586435-3054-4099-808d-8482a5449e01" path="/var/lib/kubelet/pods/67586435-3054-4099-808d-8482a5449e01/volumes" Oct 31 13:00:23.552297 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.552200 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:23.823510 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:23.823485 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:24.526086 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:24.526030 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:25.249678 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:25.249646 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb"] Oct 31 13:00:25.249862 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:25.249831 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" containerID="cri-o://c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21" gracePeriod=30 Oct 31 13:00:25.387304 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:25.387241 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:25.739086 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:25.739056 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:25.771094 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:25.771062 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:00:25.906831 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:25.906797 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:26.131932 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:26.131897 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:26.269695 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:26.269659 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:27.141217 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:27.141182 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:27.839717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:27.839687 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:30.340394 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:30.340356 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:00:30.512950 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.512927 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 13:00:30.680289 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680262 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-envoy\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680317 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-podinfo\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680372 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-credential-socket\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680425 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-data\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680507 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-kube-api-access-2wdxg\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-kube-api-access-2wdxg\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680536 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-socket\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680586 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-token\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-token\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680634 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-certs\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680886 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680688 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istiod-ca-cert\") pod \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\" (UID: \"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c\") " Oct 31 13:00:30.680886 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680799 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-credential-socket" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:30.680886 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680826 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-data" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:30.681048 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.680907 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-socket" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:30.681048 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.681008 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-certs" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:30.681246 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.681215 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istiod-ca-cert" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:30.681380 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.681243 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.681380 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.681263 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.681380 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.681272 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.681380 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.681280 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.682507 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.682487 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-envoy" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:30.682657 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.682642 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-podinfo" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:00:30.683176 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.683153 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-kube-api-access-2wdxg" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "kube-api-access-2wdxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:30.683254 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.683185 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-token" pod "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" (UID: "d6fe32e7-0e9b-4566-bb27-1d64dd1b093c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:30.782483 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.782437 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.782483 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.782459 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.782609 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.782495 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.782609 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.782504 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wdxg\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-kube-api-access-2wdxg\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:30.782609 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:30.782513 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:31.087973 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.087945 6727 generic.go:356] "Generic (PLEG): container finished" podID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerID="c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21" exitCode=0 Oct 31 13:00:31.088127 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.087993 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" Oct 31 13:00:31.088127 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.087999 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" event={"ID":"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c","Type":"ContainerDied","Data":"c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21"} Oct 31 13:00:31.088127 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.088099 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb" event={"ID":"d6fe32e7-0e9b-4566-bb27-1d64dd1b093c","Type":"ContainerDied","Data":"0933f6207ee2743dcc4d517fc128ab4f409018ffb92a1651382f6f2d51930a96"} Oct 31 13:00:31.088127 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.088120 6727 scope.go:122] "RemoveContainer" containerID="c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21" Oct 31 13:00:31.105117 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.105093 6727 scope.go:122] "RemoveContainer" containerID="c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21" Oct 31 13:00:31.105412 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:31.105388 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21\": container with ID starting with c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21 not found: ID does not exist" containerID="c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21" Oct 31 13:00:31.105610 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.105417 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21"} err="failed to get container status \"c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21\": rpc error: code = NotFound desc = could not find container \"c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21\": container with ID starting with c55f8f036382bb3e8ee678d71dad22e5f4020510b86181356f5595ae005edd21 not found: ID does not exist" Oct 31 13:00:31.117472 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.117437 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb"] Oct 31 13:00:31.121871 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.121847 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-qq4pb"] Oct 31 13:00:31.300189 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:31.300155 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" path="/var/lib/kubelet/pods/d6fe32e7-0e9b-4566-bb27-1d64dd1b093c/volumes" Oct 31 13:00:33.574622 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:33.574590 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:38.150809 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.150738 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" containerID="cri-o://2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449" gracePeriod=45 Oct 31 13:00:38.154449 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.154401 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" containerID="cri-o://272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280" gracePeriod=45 Oct 31 13:00:38.163415 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.163388 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" containerID="cri-o://fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e" gracePeriod=45 Oct 31 13:00:38.430618 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.430588 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 13:00:38.444388 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.444363 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 13:00:38.457535 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.457510 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 13:00:38.564839 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.564805 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-tmp-dir\") pod \"a0348efe-1a63-4ab5-9163-61a781568c31\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " Oct 31 13:00:38.565026 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.564857 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3720682a-e4c6-42ff-8a5b-0daf19403f66-kube-api-access-mcwj9\" (UniqueName: \"kubernetes.io/projected/3720682a-e4c6-42ff-8a5b-0daf19403f66-kube-api-access-mcwj9\") pod \"3720682a-e4c6-42ff-8a5b-0daf19403f66\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " Oct 31 13:00:38.565026 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.564885 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-home\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-home\") pod \"3720682a-e4c6-42ff-8a5b-0daf19403f66\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " Oct 31 13:00:38.565026 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.564921 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-dshm\") pod \"a22b2f57-4dc8-4add-9763-16660f67c587\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " Oct 31 13:00:38.565026 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.564971 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0348efe-1a63-4ab5-9163-61a781568c31-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0348efe-1a63-4ab5-9163-61a781568c31-tls-certs\") pod \"a0348efe-1a63-4ab5-9163-61a781568c31\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " Oct 31 13:00:38.565026 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.564997 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-dshm\") pod \"a0348efe-1a63-4ab5-9163-61a781568c31\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " Oct 31 13:00:38.565322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565058 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-home\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-home\") pod \"a22b2f57-4dc8-4add-9763-16660f67c587\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " Oct 31 13:00:38.565322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565118 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22b2f57-4dc8-4add-9763-16660f67c587-kube-api-access-wzl9q\" (UniqueName: \"kubernetes.io/projected/a22b2f57-4dc8-4add-9763-16660f67c587-kube-api-access-wzl9q\") pod \"a22b2f57-4dc8-4add-9763-16660f67c587\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " Oct 31 13:00:38.565322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565120 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-tmp-dir" pod "a0348efe-1a63-4ab5-9163-61a781568c31" (UID: "a0348efe-1a63-4ab5-9163-61a781568c31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.565322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565144 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-dshm\") pod \"3720682a-e4c6-42ff-8a5b-0daf19403f66\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " Oct 31 13:00:38.565322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565167 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-home" pod "3720682a-e4c6-42ff-8a5b-0daf19403f66" (UID: "3720682a-e4c6-42ff-8a5b-0daf19403f66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.565322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565171 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0348efe-1a63-4ab5-9163-61a781568c31-kube-api-access-bdw85\" (UniqueName: \"kubernetes.io/projected/a0348efe-1a63-4ab5-9163-61a781568c31-kube-api-access-bdw85\") pod \"a0348efe-1a63-4ab5-9163-61a781568c31\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " Oct 31 13:00:38.565322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565274 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-model-cache\") pod \"a0348efe-1a63-4ab5-9163-61a781568c31\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565332 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-home\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-home\") pod \"a0348efe-1a63-4ab5-9163-61a781568c31\" (UID: \"a0348efe-1a63-4ab5-9163-61a781568c31\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565374 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-tmp-dir\") pod \"a22b2f57-4dc8-4add-9763-16660f67c587\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565489 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-model-cache\") pod \"a22b2f57-4dc8-4add-9763-16660f67c587\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565532 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-tmp-dir\") pod \"3720682a-e4c6-42ff-8a5b-0daf19403f66\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565585 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22b2f57-4dc8-4add-9763-16660f67c587-tls-certs\" (UniqueName: \"kubernetes.io/secret/a22b2f57-4dc8-4add-9763-16660f67c587-tls-certs\") pod \"a22b2f57-4dc8-4add-9763-16660f67c587\" (UID: \"a22b2f57-4dc8-4add-9763-16660f67c587\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565619 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3720682a-e4c6-42ff-8a5b-0daf19403f66-tls-certs\" (UniqueName: \"kubernetes.io/secret/3720682a-e4c6-42ff-8a5b-0daf19403f66-tls-certs\") pod \"3720682a-e4c6-42ff-8a5b-0daf19403f66\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565630 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-home" pod "a0348efe-1a63-4ab5-9163-61a781568c31" (UID: "a0348efe-1a63-4ab5-9163-61a781568c31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565670 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-model-cache\") pod \"3720682a-e4c6-42ff-8a5b-0daf19403f66\" (UID: \"3720682a-e4c6-42ff-8a5b-0daf19403f66\") " Oct 31 13:00:38.565739 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565741 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-home" pod "a22b2f57-4dc8-4add-9763-16660f67c587" (UID: "a22b2f57-4dc8-4add-9763-16660f67c587"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565750 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-model-cache" pod "a0348efe-1a63-4ab5-9163-61a781568c31" (UID: "a0348efe-1a63-4ab5-9163-61a781568c31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.565897 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-tmp-dir" pod "3720682a-e4c6-42ff-8a5b-0daf19403f66" (UID: "3720682a-e4c6-42ff-8a5b-0daf19403f66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566110 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-model-cache" pod "a22b2f57-4dc8-4add-9763-16660f67c587" (UID: "a22b2f57-4dc8-4add-9763-16660f67c587"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566125 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-tmp-dir" pod "a22b2f57-4dc8-4add-9763-16660f67c587" (UID: "a22b2f57-4dc8-4add-9763-16660f67c587"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566188 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566209 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566223 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566237 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566249 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.566260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566260 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.566841 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566273 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.566841 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.566373 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-model-cache" pod "3720682a-e4c6-42ff-8a5b-0daf19403f66" (UID: "3720682a-e4c6-42ff-8a5b-0daf19403f66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.568267 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.568242 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-dshm" pod "a0348efe-1a63-4ab5-9163-61a781568c31" (UID: "a0348efe-1a63-4ab5-9163-61a781568c31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.568495 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.568436 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0348efe-1a63-4ab5-9163-61a781568c31-kube-api-access-bdw85" pod "a0348efe-1a63-4ab5-9163-61a781568c31" (UID: "a0348efe-1a63-4ab5-9163-61a781568c31"). InnerVolumeSpecName "kube-api-access-bdw85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:38.568745 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.568722 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-dshm" pod "a22b2f57-4dc8-4add-9763-16660f67c587" (UID: "a22b2f57-4dc8-4add-9763-16660f67c587"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.568844 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.568799 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3720682a-e4c6-42ff-8a5b-0daf19403f66-kube-api-access-mcwj9" pod "3720682a-e4c6-42ff-8a5b-0daf19403f66" (UID: "3720682a-e4c6-42ff-8a5b-0daf19403f66"). InnerVolumeSpecName "kube-api-access-mcwj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:38.568913 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.568855 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22b2f57-4dc8-4add-9763-16660f67c587-kube-api-access-wzl9q" pod "a22b2f57-4dc8-4add-9763-16660f67c587" (UID: "a22b2f57-4dc8-4add-9763-16660f67c587"). InnerVolumeSpecName "kube-api-access-wzl9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:38.568913 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.568880 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3720682a-e4c6-42ff-8a5b-0daf19403f66-tls-certs" pod "3720682a-e4c6-42ff-8a5b-0daf19403f66" (UID: "3720682a-e4c6-42ff-8a5b-0daf19403f66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:38.569192 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.569173 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-dshm" pod "3720682a-e4c6-42ff-8a5b-0daf19403f66" (UID: "3720682a-e4c6-42ff-8a5b-0daf19403f66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:38.569518 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.569498 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22b2f57-4dc8-4add-9763-16660f67c587-tls-certs" pod "a22b2f57-4dc8-4add-9763-16660f67c587" (UID: "a22b2f57-4dc8-4add-9763-16660f67c587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:38.569931 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.569911 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0348efe-1a63-4ab5-9163-61a781568c31-tls-certs" pod "a0348efe-1a63-4ab5-9163-61a781568c31" (UID: "a0348efe-1a63-4ab5-9163-61a781568c31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:38.667997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.667916 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22b2f57-4dc8-4add-9763-16660f67c587-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.667997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.667945 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3720682a-e4c6-42ff-8a5b-0daf19403f66-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.667997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.667956 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.667997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.667967 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwj9\" (UniqueName: \"kubernetes.io/projected/3720682a-e4c6-42ff-8a5b-0daf19403f66-kube-api-access-mcwj9\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.667997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.667976 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.667997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.667989 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0348efe-1a63-4ab5-9163-61a781568c31-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.667997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.668000 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0348efe-1a63-4ab5-9163-61a781568c31-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.668333 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.668008 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzl9q\" (UniqueName: \"kubernetes.io/projected/a22b2f57-4dc8-4add-9763-16660f67c587-kube-api-access-wzl9q\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.668333 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.668015 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3720682a-e4c6-42ff-8a5b-0daf19403f66-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.668333 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.668023 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdw85\" (UniqueName: \"kubernetes.io/projected/a0348efe-1a63-4ab5-9163-61a781568c31-kube-api-access-bdw85\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.668333 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.668031 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22b2f57-4dc8-4add-9763-16660f67c587-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:38.707834 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.707802 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:38.853657 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:38.853620 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:39.131552 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.131520 6727 generic.go:356] "Generic (PLEG): container finished" podID="a22b2f57-4dc8-4add-9763-16660f67c587" containerID="272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280" exitCode=0 Oct 31 13:00:39.131734 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.131567 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" Oct 31 13:00:39.131734 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.131605 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" event={"ID":"a22b2f57-4dc8-4add-9763-16660f67c587","Type":"ContainerDied","Data":"272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280"} Oct 31 13:00:39.131734 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.131648 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv" event={"ID":"a22b2f57-4dc8-4add-9763-16660f67c587","Type":"ContainerDied","Data":"82e13db46e250f22ac0e8a3390a82a863b1ef8e3add6c334e9194280dc5f1bdc"} Oct 31 13:00:39.131734 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.131670 6727 scope.go:122] "RemoveContainer" containerID="272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280" Oct 31 13:00:39.132976 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.132953 6727 generic.go:356] "Generic (PLEG): container finished" podID="a0348efe-1a63-4ab5-9163-61a781568c31" containerID="fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e" exitCode=0 Oct 31 13:00:39.133101 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.133007 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" Oct 31 13:00:39.133101 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.133030 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" event={"ID":"a0348efe-1a63-4ab5-9163-61a781568c31","Type":"ContainerDied","Data":"fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e"} Oct 31 13:00:39.133101 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.133075 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz" event={"ID":"a0348efe-1a63-4ab5-9163-61a781568c31","Type":"ContainerDied","Data":"931a2662a373e6a71ca2a16f1d63037a23cf4dc66faba9d268cb667bc9afff1c"} Oct 31 13:00:39.134657 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.134633 6727 generic.go:356] "Generic (PLEG): container finished" podID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerID="2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449" exitCode=0 Oct 31 13:00:39.134775 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.134674 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" Oct 31 13:00:39.134775 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.134674 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" event={"ID":"3720682a-e4c6-42ff-8a5b-0daf19403f66","Type":"ContainerDied","Data":"2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449"} Oct 31 13:00:39.134912 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.134780 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh" event={"ID":"3720682a-e4c6-42ff-8a5b-0daf19403f66","Type":"ContainerDied","Data":"8a969ba499ebf1108abbe17139a1e3a0d6e78a4c89f2ad7083c6b0d33258469e"} Oct 31 13:00:39.142293 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.142082 6727 scope.go:122] "RemoveContainer" containerID="272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280" Oct 31 13:00:39.142954 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:39.142718 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280\": container with ID starting with 272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280 not found: ID does not exist" containerID="272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280" Oct 31 13:00:39.142954 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.142756 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280"} err="failed to get container status \"272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280\": rpc error: code = NotFound desc = could not find container \"272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280\": container with ID starting with 272f8b98303f91b84c0758e18f80a8f47f926a8ff91c9a6984b578817983c280 not found: ID does not exist" Oct 31 13:00:39.142954 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.142776 6727 scope.go:122] "RemoveContainer" containerID="fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e" Oct 31 13:00:39.151079 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.151062 6727 scope.go:122] "RemoveContainer" containerID="fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e" Oct 31 13:00:39.151428 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:39.151330 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e\": container with ID starting with fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e not found: ID does not exist" containerID="fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e" Oct 31 13:00:39.151428 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.151354 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e"} err="failed to get container status \"fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e\": rpc error: code = NotFound desc = could not find container \"fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e\": container with ID starting with fdf9b669938cba573124e72480550607f11ddd823688030a9875e2ee563c846e not found: ID does not exist" Oct 31 13:00:39.151428 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.151375 6727 scope.go:122] "RemoveContainer" containerID="2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449" Oct 31 13:00:39.161074 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.161053 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv"] Oct 31 13:00:39.161687 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.161663 6727 scope.go:122] "RemoveContainer" containerID="2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449" Oct 31 13:00:39.161935 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:39.161917 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449\": container with ID starting with 2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449 not found: ID does not exist" containerID="2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449" Oct 31 13:00:39.161990 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.161939 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449"} err="failed to get container status \"2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449\": rpc error: code = NotFound desc = could not find container \"2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449\": container with ID starting with 2c2196204cf0e2acf7e186b6abf2d0e2674a46da076e819576d134f06670d449 not found: ID does not exist" Oct 31 13:00:39.165437 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.165415 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-w4-c67de2-kserve-7b4c565ccb-cxwvv"] Oct 31 13:00:39.184670 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.184650 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz"] Oct 31 13:00:39.188962 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.188940 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/premium-w4-c67de2-kserve-6fdd7d5557-w9gjz"] Oct 31 13:00:39.201410 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.201391 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh"] Oct 31 13:00:39.207227 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.207208 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/distinct-2-w4-c67de2-kserve-5f8fb455c5-z65kh"] Oct 31 13:00:39.302358 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.302332 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" path="/var/lib/kubelet/pods/3720682a-e4c6-42ff-8a5b-0daf19403f66/volumes" Oct 31 13:00:39.302981 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.302955 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" path="/var/lib/kubelet/pods/a0348efe-1a63-4ab5-9163-61a781568c31/volumes" Oct 31 13:00:39.303451 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.303436 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" path="/var/lib/kubelet/pods/a22b2f57-4dc8-4add-9763-16660f67c587/volumes" Oct 31 13:00:39.551007 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:39.550915 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:40.239193 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:40.238995 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:40.957535 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:40.957504 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:41.510282 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:41.510247 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8"] Oct 31 13:00:41.519367 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:41.519342 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb"] Oct 31 13:00:41.639932 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:41.639904 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:41.779076 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:41.779000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:42.608918 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:42.608883 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:43.560204 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:43.560173 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:00:43.580576 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:43.580545 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:44.208190 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:44.208160 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:44.704047 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:44.704011 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj"] Oct 31 13:00:44.704306 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:44.704278 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" podUID="10534e17-bdff-4137-a4b1-8cac04c80b73" containerName="maas-api" containerID="cri-o://8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132" gracePeriod=30 Oct 31 13:00:44.775956 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:44.775927 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:44.921388 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:44.921240 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:45.027364 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.027337 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 13:00:45.086670 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.086566 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:45.136716 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.136690 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-metrics-certs\") pod \"10534e17-bdff-4137-a4b1-8cac04c80b73\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " Oct 31 13:00:45.136892 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.136741 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10534e17-bdff-4137-a4b1-8cac04c80b73-kube-api-access-grhnr\" (UniqueName: \"kubernetes.io/projected/10534e17-bdff-4137-a4b1-8cac04c80b73-kube-api-access-grhnr\") pod \"10534e17-bdff-4137-a4b1-8cac04c80b73\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " Oct 31 13:00:45.136892 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.136858 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10534e17-bdff-4137-a4b1-8cac04c80b73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10534e17-bdff-4137-a4b1-8cac04c80b73-openshift-service-ca\") pod \"10534e17-bdff-4137-a4b1-8cac04c80b73\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " Oct 31 13:00:45.137007 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.136986 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls\") pod \"10534e17-bdff-4137-a4b1-8cac04c80b73\" (UID: \"10534e17-bdff-4137-a4b1-8cac04c80b73\") " Oct 31 13:00:45.137298 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.137264 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10534e17-bdff-4137-a4b1-8cac04c80b73-openshift-service-ca" pod "10534e17-bdff-4137-a4b1-8cac04c80b73" (UID: "10534e17-bdff-4137-a4b1-8cac04c80b73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:45.137761 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.137738 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10534e17-bdff-4137-a4b1-8cac04c80b73-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:45.139003 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.138976 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10534e17-bdff-4137-a4b1-8cac04c80b73-kube-api-access-grhnr" pod "10534e17-bdff-4137-a4b1-8cac04c80b73" (UID: "10534e17-bdff-4137-a4b1-8cac04c80b73"). InnerVolumeSpecName "kube-api-access-grhnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:45.139102 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.138999 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-metrics-certs" pod "10534e17-bdff-4137-a4b1-8cac04c80b73" (UID: "10534e17-bdff-4137-a4b1-8cac04c80b73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:45.139345 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.139329 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls" pod "10534e17-bdff-4137-a4b1-8cac04c80b73" (UID: "10534e17-bdff-4137-a4b1-8cac04c80b73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:45.174593 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.174555 6727 generic.go:356] "Generic (PLEG): container finished" podID="10534e17-bdff-4137-a4b1-8cac04c80b73" containerID="8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132" exitCode=0 Oct 31 13:00:45.174774 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.174621 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" Oct 31 13:00:45.174774 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.174638 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" event={"ID":"10534e17-bdff-4137-a4b1-8cac04c80b73","Type":"ContainerDied","Data":"8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132"} Oct 31 13:00:45.174774 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.174710 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj" event={"ID":"10534e17-bdff-4137-a4b1-8cac04c80b73","Type":"ContainerDied","Data":"4000a7dbc4bbf43a884c618806e0dc74516a4440016cb22f52237ad7827e2c2a"} Oct 31 13:00:45.174774 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.174745 6727 scope.go:122] "RemoveContainer" containerID="8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132" Oct 31 13:00:45.185264 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.185247 6727 scope.go:122] "RemoveContainer" containerID="8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132" Oct 31 13:00:45.185583 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:45.185559 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132\": container with ID starting with 8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132 not found: ID does not exist" containerID="8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132" Oct 31 13:00:45.185686 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.185593 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132"} err="failed to get container status \"8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132\": rpc error: code = NotFound desc = could not find container \"8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132\": container with ID starting with 8b366ef90466b250680f5b554384d5c41f49af3aca923871a6cf8c2de5807132 not found: ID does not exist" Oct 31 13:00:45.197105 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.197082 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj"] Oct 31 13:00:45.202888 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.202863 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c67de2-57cb84f888-b5zfj"] Oct 31 13:00:45.223964 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.223934 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:45.238459 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.238436 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:45.238459 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.238456 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10534e17-bdff-4137-a4b1-8cac04c80b73-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:45.238608 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.238494 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grhnr\" (UniqueName: \"kubernetes.io/projected/10534e17-bdff-4137-a4b1-8cac04c80b73-kube-api-access-grhnr\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:45.299388 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.299337 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10534e17-bdff-4137-a4b1-8cac04c80b73" path="/var/lib/kubelet/pods/10534e17-bdff-4137-a4b1-8cac04c80b73/volumes" Oct 31 13:00:45.577103 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.577028 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:45.937258 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:45.937131 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:46.751816 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:46.751785 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:47.025764 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:47.025693 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:47.241581 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:47.241553 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:51.741142 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:51.741109 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:51.876168 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:51.876132 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:52.010565 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:52.010485 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8"] Oct 31 13:00:52.010725 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:52.010705 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" containerID="cri-o://46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33" gracePeriod=30 Oct 31 13:00:52.078628 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:52.078597 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:53.862158 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:53.862122 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:00:56.530881 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.530790 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" containerID="cri-o://983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06" gracePeriod=45 Oct 31 13:00:56.539824 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.539797 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" containerID="cri-o://159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40" gracePeriod=45 Oct 31 13:00:56.812657 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.812627 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 13:00:56.859451 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.859413 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 13:00:56.964717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.964686 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176af21f-5986-4784-89c3-9d7b1ab15609-tls-certs\" (UniqueName: \"kubernetes.io/secret/176af21f-5986-4784-89c3-9d7b1ab15609-tls-certs\") pod \"176af21f-5986-4784-89c3-9d7b1ab15609\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " Oct 31 13:00:56.964717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.964731 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-model-cache\") pod \"176af21f-5986-4784-89c3-9d7b1ab15609\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " Oct 31 13:00:56.964966 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.964768 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-tmp-dir\") pod \"11cf690d-d2e4-426c-8b02-38a801dff8a7\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " Oct 31 13:00:56.964966 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.964796 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-model-cache\") pod \"11cf690d-d2e4-426c-8b02-38a801dff8a7\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " Oct 31 13:00:56.964966 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.964949 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176af21f-5986-4784-89c3-9d7b1ab15609-kube-api-access-ts6tr\" (UniqueName: \"kubernetes.io/projected/176af21f-5986-4784-89c3-9d7b1ab15609-kube-api-access-ts6tr\") pod \"176af21f-5986-4784-89c3-9d7b1ab15609\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " Oct 31 13:00:56.965137 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965069 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-model-cache" pod "176af21f-5986-4784-89c3-9d7b1ab15609" (UID: "176af21f-5986-4784-89c3-9d7b1ab15609"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.965137 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965075 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-model-cache" pod "11cf690d-d2e4-426c-8b02-38a801dff8a7" (UID: "11cf690d-d2e4-426c-8b02-38a801dff8a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.965137 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965113 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-tmp-dir" pod "11cf690d-d2e4-426c-8b02-38a801dff8a7" (UID: "11cf690d-d2e4-426c-8b02-38a801dff8a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.965309 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965127 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-home\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-home\") pod \"11cf690d-d2e4-426c-8b02-38a801dff8a7\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " Oct 31 13:00:56.965309 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965200 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-dshm\") pod \"11cf690d-d2e4-426c-8b02-38a801dff8a7\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " Oct 31 13:00:56.965309 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965280 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-tmp-dir\") pod \"176af21f-5986-4784-89c3-9d7b1ab15609\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " Oct 31 13:00:56.965479 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965316 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-home\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-home\") pod \"176af21f-5986-4784-89c3-9d7b1ab15609\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " Oct 31 13:00:56.965479 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965322 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-home" pod "11cf690d-d2e4-426c-8b02-38a801dff8a7" (UID: "11cf690d-d2e4-426c-8b02-38a801dff8a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.965479 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965366 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-dshm\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-dshm\") pod \"176af21f-5986-4784-89c3-9d7b1ab15609\" (UID: \"176af21f-5986-4784-89c3-9d7b1ab15609\") " Oct 31 13:00:56.965479 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965430 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11cf690d-d2e4-426c-8b02-38a801dff8a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/11cf690d-d2e4-426c-8b02-38a801dff8a7-tls-certs\") pod \"11cf690d-d2e4-426c-8b02-38a801dff8a7\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " Oct 31 13:00:56.965690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965503 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cf690d-d2e4-426c-8b02-38a801dff8a7-kube-api-access-fz8q2\" (UniqueName: \"kubernetes.io/projected/11cf690d-d2e4-426c-8b02-38a801dff8a7-kube-api-access-fz8q2\") pod \"11cf690d-d2e4-426c-8b02-38a801dff8a7\" (UID: \"11cf690d-d2e4-426c-8b02-38a801dff8a7\") " Oct 31 13:00:56.965690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965525 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-tmp-dir" pod "176af21f-5986-4784-89c3-9d7b1ab15609" (UID: "176af21f-5986-4784-89c3-9d7b1ab15609"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.965690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965617 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-home" pod "176af21f-5986-4784-89c3-9d7b1ab15609" (UID: "176af21f-5986-4784-89c3-9d7b1ab15609"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.965950 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965923 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:56.966086 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965953 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:56.966086 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965969 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:56.966086 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965984 6727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-tmp-dir\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:56.966086 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.965996 6727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-home\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:56.966086 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.966009 6727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-model-cache\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:56.967675 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.967622 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176af21f-5986-4784-89c3-9d7b1ab15609-tls-certs" pod "176af21f-5986-4784-89c3-9d7b1ab15609" (UID: "176af21f-5986-4784-89c3-9d7b1ab15609"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:56.967675 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.967629 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176af21f-5986-4784-89c3-9d7b1ab15609-kube-api-access-ts6tr" pod "176af21f-5986-4784-89c3-9d7b1ab15609" (UID: "176af21f-5986-4784-89c3-9d7b1ab15609"). InnerVolumeSpecName "kube-api-access-ts6tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:56.967675 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.967649 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-dshm" pod "176af21f-5986-4784-89c3-9d7b1ab15609" (UID: "176af21f-5986-4784-89c3-9d7b1ab15609"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.967675 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.967627 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-dshm" pod "11cf690d-d2e4-426c-8b02-38a801dff8a7" (UID: "11cf690d-d2e4-426c-8b02-38a801dff8a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:56.968133 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.968105 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cf690d-d2e4-426c-8b02-38a801dff8a7-tls-certs" pod "11cf690d-d2e4-426c-8b02-38a801dff8a7" (UID: "11cf690d-d2e4-426c-8b02-38a801dff8a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:56.968214 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:56.968111 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cf690d-d2e4-426c-8b02-38a801dff8a7-kube-api-access-fz8q2" pod "11cf690d-d2e4-426c-8b02-38a801dff8a7" (UID: "11cf690d-d2e4-426c-8b02-38a801dff8a7"). InnerVolumeSpecName "kube-api-access-fz8q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:57.066979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.066902 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176af21f-5986-4784-89c3-9d7b1ab15609-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.066979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.066934 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cf690d-d2e4-426c-8b02-38a801dff8a7-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.066979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.066946 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8q2\" (UniqueName: \"kubernetes.io/projected/11cf690d-d2e4-426c-8b02-38a801dff8a7-kube-api-access-fz8q2\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.066979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.066955 6727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176af21f-5986-4784-89c3-9d7b1ab15609-tls-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.066979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.066964 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6tr\" (UniqueName: \"kubernetes.io/projected/176af21f-5986-4784-89c3-9d7b1ab15609-kube-api-access-ts6tr\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.066979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.066972 6727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cf690d-d2e4-426c-8b02-38a801dff8a7-dshm\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.248916 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.248892 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 13:00:57.253155 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.253129 6727 generic.go:356] "Generic (PLEG): container finished" podID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerID="46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33" exitCode=0 Oct 31 13:00:57.253294 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.253164 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" Oct 31 13:00:57.253294 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.253207 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" event={"ID":"6bff0de6-37e4-4e89-b0e8-fd7a153c0098","Type":"ContainerDied","Data":"46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33"} Oct 31 13:00:57.253294 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.253286 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8" event={"ID":"6bff0de6-37e4-4e89-b0e8-fd7a153c0098","Type":"ContainerDied","Data":"82db206ca5a2dff1d05f03b95bc45ac3e617a0cc6c0ce102662587625ed9d8a9"} Oct 31 13:00:57.253425 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.253311 6727 scope.go:122] "RemoveContainer" containerID="46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33" Oct 31 13:00:57.254638 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.254618 6727 generic.go:356] "Generic (PLEG): container finished" podID="176af21f-5986-4784-89c3-9d7b1ab15609" containerID="983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06" exitCode=0 Oct 31 13:00:57.254758 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.254650 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" event={"ID":"176af21f-5986-4784-89c3-9d7b1ab15609","Type":"ContainerDied","Data":"983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06"} Oct 31 13:00:57.254758 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.254679 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" Oct 31 13:00:57.254758 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.254689 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8" event={"ID":"176af21f-5986-4784-89c3-9d7b1ab15609","Type":"ContainerDied","Data":"d43a3df7f58d851f153acf04ead95382b84f53a6a617533704f92c7844546814"} Oct 31 13:00:57.256157 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.256138 6727 generic.go:356] "Generic (PLEG): container finished" podID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerID="159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40" exitCode=0 Oct 31 13:00:57.256258 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.256169 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" Oct 31 13:00:57.256258 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.256223 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" event={"ID":"11cf690d-d2e4-426c-8b02-38a801dff8a7","Type":"ContainerDied","Data":"159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40"} Oct 31 13:00:57.256394 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.256258 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb" event={"ID":"11cf690d-d2e4-426c-8b02-38a801dff8a7","Type":"ContainerDied","Data":"8ab3b79287780b7aee6d32b961a1ffa1e4107e323087bfaf2af483d5f3c95dc9"} Oct 31 13:00:57.264532 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.264516 6727 scope.go:122] "RemoveContainer" containerID="46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33" Oct 31 13:00:57.264796 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:57.264776 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33\": container with ID starting with 46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33 not found: ID does not exist" containerID="46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33" Oct 31 13:00:57.264873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.264806 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33"} err="failed to get container status \"46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33\": rpc error: code = NotFound desc = could not find container \"46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33\": container with ID starting with 46cb4dac83e4590fdbd0d40b0eba5a9b34cd9160cd5b456ed0abdc7ff6bd4e33 not found: ID does not exist" Oct 31 13:00:57.264873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.264828 6727 scope.go:122] "RemoveContainer" containerID="983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06" Oct 31 13:00:57.273306 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.273284 6727 scope.go:122] "RemoveContainer" containerID="983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06" Oct 31 13:00:57.273598 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:57.273580 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06\": container with ID starting with 983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06 not found: ID does not exist" containerID="983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06" Oct 31 13:00:57.273676 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.273604 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06"} err="failed to get container status \"983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06\": rpc error: code = NotFound desc = could not find container \"983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06\": container with ID starting with 983354a192e4308fc3decb562c7327ecc8b98b95828a3abb1815d6cd64824a06 not found: ID does not exist" Oct 31 13:00:57.273676 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.273621 6727 scope.go:122] "RemoveContainer" containerID="159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40" Oct 31 13:00:57.309732 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.309713 6727 scope.go:122] "RemoveContainer" containerID="159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40" Oct 31 13:00:57.310088 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:00:57.310068 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40\": container with ID starting with 159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40 not found: ID does not exist" containerID="159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40" Oct 31 13:00:57.310198 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.310094 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40"} err="failed to get container status \"159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40\": rpc error: code = NotFound desc = could not find container \"159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40\": container with ID starting with 159fe2119829a20042e64d260e36c186cff7c87d6488e531a768cde4cbe8ec40 not found: ID does not exist" Oct 31 13:00:57.310198 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.310106 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8"] Oct 31 13:00:57.313814 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.313794 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-w6-67d410-kserve-65d4d57-9skn8"] Oct 31 13:00:57.324240 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.324218 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb"] Oct 31 13:00:57.337088 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.337062 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/premium-w6-67d410-kserve-7f8c876c7d-hchhb"] Oct 31 13:00:57.370971 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.370946 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-podinfo\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371107 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.370991 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-token\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-token\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371107 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371049 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istiod-ca-cert\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371107 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371093 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-kube-api-access-9vggn\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-kube-api-access-9vggn\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371274 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371131 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-data\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371274 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371242 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-envoy\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371406 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371292 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-socket\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371511 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371492 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-credential-socket\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371576 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371543 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-certs\") pod \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\" (UID: \"6bff0de6-37e4-4e89-b0e8-fd7a153c0098\") " Oct 31 13:00:57.371652 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371620 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-data" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:57.372195 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371670 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istiod-ca-cert" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:57.372195 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.371878 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-credential-socket" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:57.372717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.372013 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-socket" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:57.372717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.372167 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-certs" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:57.372717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.372510 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.372717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.372533 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.372717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.372548 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.373385 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.373357 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-podinfo" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:00:57.373492 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.373430 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-kube-api-access-9vggn" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "kube-api-access-9vggn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:57.373624 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.373608 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-token" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:57.373919 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.373904 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-envoy" pod "6bff0de6-37e4-4e89-b0e8-fd7a153c0098" (UID: "6bff0de6-37e4-4e89-b0e8-fd7a153c0098"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:57.473643 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.473607 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vggn\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-kube-api-access-9vggn\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.473643 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.473643 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.473847 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.473661 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.473847 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.473676 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.473847 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.473688 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.473847 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.473698 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bff0de6-37e4-4e89-b0e8-fd7a153c0098-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:57.575271 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.575207 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8"] Oct 31 13:00:57.581842 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:57.581817 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-j2dl8"] Oct 31 13:00:59.300143 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:59.300109 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" path="/var/lib/kubelet/pods/11cf690d-d2e4-426c-8b02-38a801dff8a7/volumes" Oct 31 13:00:59.300599 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:59.300583 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" path="/var/lib/kubelet/pods/176af21f-5986-4784-89c3-9d7b1ab15609/volumes" Oct 31 13:00:59.301050 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:59.301020 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" path="/var/lib/kubelet/pods/6bff0de6-37e4-4e89-b0e8-fd7a153c0098/volumes" Oct 31 13:00:59.453407 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:59.453377 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:59.592245 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:59.592195 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:59.737526 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:00:59.737459 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:00.342996 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:01:00.342953 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:01:01.111411 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:01.111374 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:02.490446 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:02.490416 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:02.643770 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:02.643732 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:01:02.649542 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:02.649516 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:02.783022 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:02.782925 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:03.246965 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:03.246915 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:07.272199 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:07.272166 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:07.940774 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:07.940739 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:01:09.014636 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:09.014603 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:09.158986 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:09.158949 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:09.303356 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:09.303268 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:09.450607 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:09.450571 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc"] Oct 31 13:01:09.450875 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:09.450828 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" containerID="cri-o://1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5" gracePeriod=30 Oct 31 13:01:09.527510 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:09.527459 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:12.235423 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.235388 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d9577675-zr4jx"] Oct 31 13:01:12.235865 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.235617 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64d9577675-zr4jx" podUID="5c4ced04-45b1-440a-87c2-e0d5f595981a" containerName="manager" containerID="cri-o://f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd" gracePeriod=30 Oct 31 13:01:12.488523 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.488448 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 13:01:12.530020 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.529974 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4ced04-45b1-440a-87c2-e0d5f595981a-kube-api-access-nrmj2\" (UniqueName: \"kubernetes.io/projected/5c4ced04-45b1-440a-87c2-e0d5f595981a-kube-api-access-nrmj2\") pod \"5c4ced04-45b1-440a-87c2-e0d5f595981a\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " Oct 31 13:01:12.530210 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.530146 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-metrics-certs\") pod \"5c4ced04-45b1-440a-87c2-e0d5f595981a\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " Oct 31 13:01:12.530288 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.530231 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-webhook-cert\") pod \"5c4ced04-45b1-440a-87c2-e0d5f595981a\" (UID: \"5c4ced04-45b1-440a-87c2-e0d5f595981a\") " Oct 31 13:01:12.532861 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.532835 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-webhook-cert" pod "5c4ced04-45b1-440a-87c2-e0d5f595981a" (UID: "5c4ced04-45b1-440a-87c2-e0d5f595981a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:01:12.532986 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.532923 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4ced04-45b1-440a-87c2-e0d5f595981a-kube-api-access-nrmj2" pod "5c4ced04-45b1-440a-87c2-e0d5f595981a" (UID: "5c4ced04-45b1-440a-87c2-e0d5f595981a"). InnerVolumeSpecName "kube-api-access-nrmj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:12.536560 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.536534 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-metrics-certs" pod "5c4ced04-45b1-440a-87c2-e0d5f595981a" (UID: "5c4ced04-45b1-440a-87c2-e0d5f595981a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:01:12.631728 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.631693 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:12.631728 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.631726 6727 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c4ced04-45b1-440a-87c2-e0d5f595981a-webhook-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:12.631938 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:12.631740 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrmj2\" (UniqueName: \"kubernetes.io/projected/5c4ced04-45b1-440a-87c2-e0d5f595981a-kube-api-access-nrmj2\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:13.355474 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.355438 6727 generic.go:356] "Generic (PLEG): container finished" podID="5c4ced04-45b1-440a-87c2-e0d5f595981a" containerID="f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd" exitCode=0 Oct 31 13:01:13.355981 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.355505 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-zr4jx" Oct 31 13:01:13.355981 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.355538 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-zr4jx" event={"ID":"5c4ced04-45b1-440a-87c2-e0d5f595981a","Type":"ContainerDied","Data":"f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd"} Oct 31 13:01:13.355981 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.355584 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-zr4jx" event={"ID":"5c4ced04-45b1-440a-87c2-e0d5f595981a","Type":"ContainerDied","Data":"93b9a43d55882416e8b8a60566bd5fc3f12b8aa75b92581d05153e1d96e8669e"} Oct 31 13:01:13.355981 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.355606 6727 scope.go:122] "RemoveContainer" containerID="f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd" Oct 31 13:01:13.365745 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.365722 6727 scope.go:122] "RemoveContainer" containerID="f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd" Oct 31 13:01:13.366093 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:01:13.366068 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd\": container with ID starting with f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd not found: ID does not exist" containerID="f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd" Oct 31 13:01:13.366174 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.366108 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd"} err="failed to get container status \"f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd\": rpc error: code = NotFound desc = could not find container \"f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd\": container with ID starting with f30490cba4d773ff6cedfb2a86228109d512bda356afad0ebb5587643557ccbd not found: ID does not exist" Oct 31 13:01:13.375045 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.374983 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d9577675-zr4jx"] Oct 31 13:01:13.377237 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:13.377205 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64d9577675-zr4jx"] Oct 31 13:01:14.762578 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.762538 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 13:01:14.854949 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.854907 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-envoy\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855129 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.854991 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-token\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855129 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855014 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-podinfo\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855129 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855064 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-certs\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855276 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855185 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6b188d-78b4-495c-937a-9fc6e8150593-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b188d-78b4-495c-937a-9fc6e8150593-istiod-ca-cert\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855276 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855236 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-kube-api-access-4smdh\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-kube-api-access-4smdh\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855391 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855346 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-socket\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855391 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855363 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-certs" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:14.855546 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855405 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-data\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855546 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855442 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-credential-socket\") pod \"ff6b188d-78b4-495c-937a-9fc6e8150593\" (UID: \"ff6b188d-78b4-495c-937a-9fc6e8150593\") " Oct 31 13:01:14.855711 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855605 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6b188d-78b4-495c-937a-9fc6e8150593-istiod-ca-cert" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:14.855711 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855689 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-socket" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:14.855711 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855674 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-data" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:14.855914 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855882 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-credential-socket" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:14.855995 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855948 6727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.855995 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855969 6727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b188d-78b4-495c-937a-9fc6e8150593-istiod-ca-cert\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.855995 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855984 6727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-workload-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.856106 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.855998 6727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-data\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.857253 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.857235 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-token" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:14.857639 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.857599 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-podinfo" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:01:14.857709 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.857639 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-envoy" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:14.857709 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.857694 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-kube-api-access-4smdh" pod "ff6b188d-78b4-495c-937a-9fc6e8150593" (UID: "ff6b188d-78b4-495c-937a-9fc6e8150593"). InnerVolumeSpecName "kube-api-access-4smdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:14.956819 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.956779 6727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-token\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.956819 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.956816 6727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-podinfo\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.956819 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.956826 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smdh\" (UniqueName: \"kubernetes.io/projected/ff6b188d-78b4-495c-937a-9fc6e8150593-kube-api-access-4smdh\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.957071 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.956837 6727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-credential-socket\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:14.957071 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:14.956846 6727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b188d-78b4-495c-937a-9fc6e8150593-istio-envoy\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:15.304765 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.304726 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4ced04-45b1-440a-87c2-e0d5f595981a" path="/var/lib/kubelet/pods/5c4ced04-45b1-440a-87c2-e0d5f595981a/volumes" Oct 31 13:01:15.370147 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.370102 6727 generic.go:356] "Generic (PLEG): container finished" podID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerID="1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5" exitCode=0 Oct 31 13:01:15.370330 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.370232 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" event={"ID":"ff6b188d-78b4-495c-937a-9fc6e8150593","Type":"ContainerDied","Data":"1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5"} Oct 31 13:01:15.370330 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.370286 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" event={"ID":"ff6b188d-78b4-495c-937a-9fc6e8150593","Type":"ContainerDied","Data":"3481dd8d992196bd639b08de26ac598a1ef2c64c0395f7b1f40702765188d2a6"} Oct 31 13:01:15.370330 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.370319 6727 scope.go:122] "RemoveContainer" containerID="1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5" Oct 31 13:01:15.370604 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.370443 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc" Oct 31 13:01:15.387821 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.387690 6727 scope.go:122] "RemoveContainer" containerID="1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5" Oct 31 13:01:15.392652 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.392618 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc"] Oct 31 13:01:15.393065 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:01:15.393028 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5\": container with ID starting with 1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5 not found: ID does not exist" containerID="1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5" Oct 31 13:01:15.393198 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.393079 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5"} err="failed to get container status \"1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5\": rpc error: code = NotFound desc = could not find container \"1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5\": container with ID starting with 1752431c9697029746ea55fef2d78fab2814fb0f448abd92ee5c6e930c1f91f5 not found: ID does not exist" Oct 31 13:01:15.401775 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:15.401740 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rkgcc"] Oct 31 13:01:17.299199 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:17.299164 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" path="/var/lib/kubelet/pods/ff6b188d-78b4-495c-937a-9fc6e8150593/volumes" Oct 31 13:01:18.474876 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:18.474835 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:20.341217 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:20.341144 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:25.156868 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:25.156827 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:30.340932 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:01:30.340888 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:01:45.856318 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:45.856269 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:47.156664 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.156632 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d"] Oct 31 13:01:47.157231 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157210 6727 cpu_manager.go:414] "containerMap: removing container" podUID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" containerName="payload-pre-processing" Oct 31 13:01:47.157340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157234 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" containerName="payload-pre-processing" Oct 31 13:01:47.157340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157256 6727 cpu_manager.go:414] "containerMap: removing container" podUID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" containerName="payload-processing" Oct 31 13:01:47.157340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157264 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" containerName="payload-processing" Oct 31 13:01:47.157340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157307 6727 cpu_manager.go:414] "containerMap: removing container" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" Oct 31 13:01:47.157340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157315 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" Oct 31 13:01:47.157340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157333 6727 cpu_manager.go:414] "containerMap: removing container" podUID="2e47b7b7-859b-4806-8829-b41130e55cb3" containerName="payload-pre-processing" Oct 31 13:01:47.157340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157341 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e47b7b7-859b-4806-8829-b41130e55cb3" containerName="payload-pre-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157357 6727 cpu_manager.go:414] "containerMap: removing container" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157364 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157379 6727 cpu_manager.go:414] "containerMap: removing container" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157387 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157401 6727 cpu_manager.go:414] "containerMap: removing container" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157409 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157419 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" containerName="maas-api" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157426 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" containerName="maas-api" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157438 6727 cpu_manager.go:414] "containerMap: removing container" podUID="995e8461-e024-4e86-899d-b5cf67abb1e0" containerName="payload-pre-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157446 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995e8461-e024-4e86-899d-b5cf67abb1e0" containerName="payload-pre-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157457 6727 cpu_manager.go:414] "containerMap: removing container" podUID="dd401d12-5957-4561-93ec-71b6a56e614b" containerName="payload-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157496 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd401d12-5957-4561-93ec-71b6a56e614b" containerName="payload-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157507 6727 cpu_manager.go:414] "containerMap: removing container" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157514 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157524 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f28e1040-0b3d-45c3-aa1b-80a1856a3e10" containerName="payload-pre-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157530 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28e1040-0b3d-45c3-aa1b-80a1856a3e10" containerName="payload-pre-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157545 6727 cpu_manager.go:414] "containerMap: removing container" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157552 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157562 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5c4ced04-45b1-440a-87c2-e0d5f595981a" containerName="manager" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157568 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4ced04-45b1-440a-87c2-e0d5f595981a" containerName="manager" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157578 6727 cpu_manager.go:414] "containerMap: removing container" podUID="b0076d52-4c70-4355-b8f1-7c90781701b6" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157585 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0076d52-4c70-4355-b8f1-7c90781701b6" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157603 6727 cpu_manager.go:414] "containerMap: removing container" podUID="888b8db5-5f7a-49f1-91a7-7ddabdd54f00" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157610 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888b8db5-5f7a-49f1-91a7-7ddabdd54f00" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157620 6727 cpu_manager.go:414] "containerMap: removing container" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157628 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157639 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157647 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157658 6727 cpu_manager.go:414] "containerMap: removing container" podUID="25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" containerName="payload-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157665 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" containerName="payload-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157681 6727 cpu_manager.go:414] "containerMap: removing container" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157688 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157699 6727 cpu_manager.go:414] "containerMap: removing container" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157706 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157717 6727 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157724 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157735 6727 cpu_manager.go:414] "containerMap: removing container" podUID="e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" containerName="maas-api" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157743 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" containerName="maas-api" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157754 6727 cpu_manager.go:414] "containerMap: removing container" podUID="48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" containerName="payload-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157761 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" containerName="payload-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157772 6727 cpu_manager.go:414] "containerMap: removing container" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157779 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157790 6727 cpu_manager.go:414] "containerMap: removing container" podUID="1d707124-abc0-4daa-8a91-0af9e6081951" containerName="payload-processing" Oct 31 13:01:47.157781 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157797 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d707124-abc0-4daa-8a91-0af9e6081951" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157806 6727 cpu_manager.go:414] "containerMap: removing container" podUID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157813 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157823 6727 cpu_manager.go:414] "containerMap: removing container" podUID="ab740d09-6da9-4783-b693-8e6d88b8b5ee" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157832 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab740d09-6da9-4783-b693-8e6d88b8b5ee" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157844 6727 cpu_manager.go:414] "containerMap: removing container" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157851 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157863 6727 cpu_manager.go:414] "containerMap: removing container" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157870 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157880 6727 cpu_manager.go:414] "containerMap: removing container" podUID="10534e17-bdff-4137-a4b1-8cac04c80b73" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157887 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10534e17-bdff-4137-a4b1-8cac04c80b73" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157897 6727 cpu_manager.go:414] "containerMap: removing container" podUID="badf7b01-8d02-47e5-877f-addd475b2921" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157904 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf7b01-8d02-47e5-877f-addd475b2921" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157919 6727 cpu_manager.go:414] "containerMap: removing container" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157926 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157939 6727 cpu_manager.go:414] "containerMap: removing container" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157946 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157955 6727 cpu_manager.go:414] "containerMap: removing container" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157963 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157972 6727 cpu_manager.go:414] "containerMap: removing container" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157979 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157988 6727 cpu_manager.go:414] "containerMap: removing container" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.157995 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158008 6727 cpu_manager.go:414] "containerMap: removing container" podUID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158021 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158032 6727 cpu_manager.go:414] "containerMap: removing container" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158040 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158053 6727 cpu_manager.go:414] "containerMap: removing container" podUID="c60a1c48-7c98-4f57-bda4-1d5aa6be1273" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158059 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60a1c48-7c98-4f57-bda4-1d5aa6be1273" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158071 6727 cpu_manager.go:414] "containerMap: removing container" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158077 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158087 6727 cpu_manager.go:414] "containerMap: removing container" podUID="fda97a76-3a20-4ee9-9a41-dfdc3887df90" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158093 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda97a76-3a20-4ee9-9a41-dfdc3887df90" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158106 6727 cpu_manager.go:414] "containerMap: removing container" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158114 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158122 6727 cpu_manager.go:414] "containerMap: removing container" podUID="7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158129 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158237 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fdb1c5f-7bad-47ae-a852-e892a74f5cfa" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158254 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5a92fb-09f8-4caf-8a18-652b94bfa460" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158263 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8e0968-6f24-450b-8f8f-e7b00f1fb768" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158276 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e47b7b7-859b-4806-8829-b41130e55cb3" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158286 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad04ba7-e99f-4779-95fd-e0ebd5e889ce" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158296 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158305 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="095de43a-2809-4ebe-aaf1-208d47b55993" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158316 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6faedd5-b9c9-47b6-bcdd-d2c5729f9fc0" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158326 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="995e8461-e024-4e86-899d-b5cf67abb1e0" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158335 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd401d12-5957-4561-93ec-71b6a56e614b" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158347 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="07021190-faf9-432c-b168-573737c7a3c4" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158356 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28e1040-0b3d-45c3-aa1b-80a1856a3e10" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158367 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="67586435-3054-4099-808d-8482a5449e01" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158378 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4ced04-45b1-440a-87c2-e0d5f595981a" containerName="manager" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158386 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0076d52-4c70-4355-b8f1-7c90781701b6" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158396 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="888b8db5-5f7a-49f1-91a7-7ddabdd54f00" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158407 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="62233c53-e636-4b2d-b52f-2b9959ee47fd" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158415 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45cda06-0d2f-45f4-917f-e854e4cb30ba" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158423 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="25da6fc3-3543-4a22-bcb5-efbc8f5f1a21" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158433 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c629cd-c1e0-4fbd-8b03-1eea66a6dc16" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158442 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5105fe-dcce-40a2-9cb9-71cad6ea8ff6" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158451 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b188d-78b4-495c-937a-9fc6e8150593" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158475 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f24a9a-03ff-49d8-ae6a-b47c7a6791d6" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158483 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="48eb36e6-90c4-4d4c-97a7-aea1a48a7e4e" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158492 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3720682a-e4c6-42ff-8a5b-0daf19403f66" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158500 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d707124-abc0-4daa-8a91-0af9e6081951" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158509 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc0a033-4793-43ba-bb55-a960b69cf7dc" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158519 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab740d09-6da9-4783-b693-8e6d88b8b5ee" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158526 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158536 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0348efe-1a63-4ab5-9163-61a781568c31" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158546 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="10534e17-bdff-4137-a4b1-8cac04c80b73" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158555 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf7b01-8d02-47e5-877f-addd475b2921" containerName="payload-pre-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158564 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fe32e7-0e9b-4566-bb27-1d64dd1b093c" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158574 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="543a8bb9-7cd3-4055-8109-7fb9d60590a8" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158582 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22b2f57-4dc8-4add-9763-16660f67c587" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158590 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2faa2b8a-979b-4b84-a35a-32d9b7413e75" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158600 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="441559e3-5ae0-40bb-b7cc-604837dea088" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158609 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb6e92d-9d0a-41fc-bd1f-8ace2848716e" containerName="payload-processing" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158618 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158626 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="176af21f-5986-4784-89c3-9d7b1ab15609" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158635 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60a1c48-7c98-4f57-bda4-1d5aa6be1273" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158645 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bff0de6-37e4-4e89-b0e8-fd7a153c0098" containerName="istio-proxy" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158654 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda97a76-3a20-4ee9-9a41-dfdc3887df90" containerName="maas-api" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158663 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cf690d-d2e4-426c-8b02-38a801dff8a7" containerName="main" Oct 31 13:01:47.160123 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.158670 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7543a97c-b2c1-46ef-82bc-57bb1ecd2e67" containerName="maas-api" Oct 31 13:01:47.163207 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.161856 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168313 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168290 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6630a4-05af-4601-bc6c-4eced69824f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168442 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168342 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9m2l\" (UniqueName: \"kubernetes.io/projected/1e6630a4-05af-4601-bc6c-4eced69824f0-kube-api-access-k9m2l\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168442 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168368 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168442 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168401 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168671 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168481 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168671 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168516 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168810 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168751 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168810 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168799 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.168929 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.168834 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.181346 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.181319 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d"] Oct 31 13:01:47.269535 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269509 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.269701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269551 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.269701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269575 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.269701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269612 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6630a4-05af-4601-bc6c-4eced69824f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.269701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269650 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9m2l\" (UniqueName: \"kubernetes.io/projected/1e6630a4-05af-4601-bc6c-4eced69824f0-kube-api-access-k9m2l\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.269701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269672 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.269701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269694 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.270052 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269757 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.270052 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.269781 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.270219 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.270196 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.270300 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.270256 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.270362 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.270317 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.270538 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.270512 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6630a4-05af-4601-bc6c-4eced69824f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.270779 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.270758 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.272212 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.272187 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.272363 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.272341 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.283041 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.283016 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e6630a4-05af-4601-bc6c-4eced69824f0-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.283194 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.283175 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9m2l\" (UniqueName: \"kubernetes.io/projected/1e6630a4-05af-4601-bc6c-4eced69824f0-kube-api-access-k9m2l\") pod \"maas-default-gateway-openshift-default-645cf464d7-87q4d\" (UID: \"1e6630a4-05af-4601-bc6c-4eced69824f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.478797 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.478707 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:47.759439 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.759369 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d"] Oct 31 13:01:47.765909 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:01:47.765875 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6630a4_05af_4601_bc6c_4eced69824f0.slice/crio-10c9ae458c77c77d1d5248c779963c724378eb1aae997c29e49d16585fd680b4 WatchSource:0}: Error finding container 10c9ae458c77c77d1d5248c779963c724378eb1aae997c29e49d16585fd680b4: Status 404 returned error can't find the container with id 10c9ae458c77c77d1d5248c779963c724378eb1aae997c29e49d16585fd680b4 Oct 31 13:01:47.767561 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.767525 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 13:01:47.767674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.767594 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 13:01:47.767674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:47.767621 6727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 13:01:48.176662 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:48.176627 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:48.528322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:48.528230 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" event={"ID":"1e6630a4-05af-4601-bc6c-4eced69824f0","Type":"ContainerStarted","Data":"1d2e43d5428ce8f127a0a3b8276325702317bc832538a732699c02d5d71c1a1d"} Oct 31 13:01:48.528322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:48.528285 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" event={"ID":"1e6630a4-05af-4601-bc6c-4eced69824f0","Type":"ContainerStarted","Data":"10c9ae458c77c77d1d5248c779963c724378eb1aae997c29e49d16585fd680b4"} Oct 31 13:01:48.553036 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:48.552989 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" podStartSLOduration=1.552977 podStartE2EDuration="1.552977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:48.55067921 +0000 UTC m=+2019.365693645" watchObservedRunningTime="2026-10-31 13:01:48.552977 +0000 UTC m=+2019.367991390" Oct 31 13:01:49.489487 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:49.488536 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:49.489989 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:49.489841 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-87q4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 13:01:49.489989 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:49.489890 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" podUID="1e6630a4-05af-4601-bc6c-4eced69824f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 13:01:50.487489 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:50.485080 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-87q4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 13:01:50.487489 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:50.485149 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" podUID="1e6630a4-05af-4601-bc6c-4eced69824f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 13:01:51.487875 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:51.487838 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-87q4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 13:01:51.488330 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:51.487902 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" podUID="1e6630a4-05af-4601-bc6c-4eced69824f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 13:01:52.487486 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:52.485908 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-87q4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 13:01:52.487486 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:52.485966 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" podUID="1e6630a4-05af-4601-bc6c-4eced69824f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 13:01:53.480585 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:53.480550 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-87q4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 13:01:53.481117 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:53.481091 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" podUID="1e6630a4-05af-4601-bc6c-4eced69824f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 13:01:54.479091 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:54.479056 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-87q4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 13:01:54.479271 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:54.479109 6727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" podUID="1e6630a4-05af-4601-bc6c-4eced69824f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 13:01:55.859292 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:55.859261 6727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:55.859834 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:55.859648 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:01:55.860501 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:01:55.860479 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-87q4d" Oct 31 13:02:00.343536 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:02:00.343495 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:02:02.251081 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:02.251046 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:02.395672 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:02.395639 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:09.065900 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:09.065854 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:09.333371 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:09.333340 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:21.647264 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:21.647233 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:02:24.159606 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:24.159569 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:02:24.160013 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:24.159639 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:02:30.341222 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:02:30.341183 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:02:39.860612 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:39.860578 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:02:44.356383 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:44.356348 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:02:58.641593 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:02:58.641563 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:03:00.340994 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:03:00.340960 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:03:09.119481 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:09.119438 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:09.386967 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:09.386889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:11.653912 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:11.653878 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:03:16.299220 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:16.299187 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:16.428628 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:16.428600 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:30.340702 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:03:30.340662 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:03:35.569946 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:35.569909 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:03:38.927515 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:38.927456 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:46.029552 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:46.029514 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:03:50.309605 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:03:50.309573 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:04:00.341217 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:04:00.341178 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:04:06.421438 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:04:06.421399 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:04:09.188005 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:04:09.187964 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:10.584347 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:04:10.584294 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:04:24.454952 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:04:24.454863 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:04:29.641744 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:04:29.641705 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:04:30.341442 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:04:30.341407 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:04:43.252931 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:04:43.252894 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:04:47.332506 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:04:47.331543 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:05:00.342052 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:05:00.342008 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:05:00.817229 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:00.817146 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:00.968171 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:00.968142 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:05:09.236008 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:09.235974 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:11.745271 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:11.745239 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:05:16.745362 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:16.745324 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:05:21.488560 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:21.488508 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:30.342794 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:05:30.342759 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:05:33.246843 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:33.246808 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:05:36.937892 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:36.937851 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:05:42.755523 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:05:42.755478 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:06:00.341178 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:06:00.341142 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:06:00.866889 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:00.866860 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:06:09.306158 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:09.306120 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:11.217822 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:11.217788 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:06:26.427068 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:26.427030 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:06:30.341817 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:06:30.341776 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:06:30.932047 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:30.932014 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:06:33.819400 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:33.819357 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:34.975581 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:34.975550 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:37.409664 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:37.409629 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:37.589069 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:37.589041 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:38.725195 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:38.725169 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:38.906839 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:38.906781 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:39.653486 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:39.650649 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:40.551495 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:40.546862 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:41.237001 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:41.236964 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:41.440340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:41.440311 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:06:42.031178 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:42.031148 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:42.730354 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:42.730272 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:43.439119 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:43.439084 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:44.137407 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:44.137375 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:44.928783 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:44.928751 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:45.640694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:45.640658 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:46.332346 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:46.332315 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:47.162537 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:47.161907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:47.956319 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:47.956282 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:48.632478 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:48.632436 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:49.353184 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:49.353148 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:50.142000 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:50.141743 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:51.170363 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:51.170325 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:51.729855 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:51.729692 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:52.313842 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:52.313813 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:52.533542 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:52.533509 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:52.881605 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:52.881573 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:53.030214 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:53.030182 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:53.195144 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:53.195032 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:53.364836 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:53.364808 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:54.010714 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:54.010680 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:54.818833 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:54.818793 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:55.269710 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:55.269661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:55.639773 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:55.639737 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:56.347992 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:56.347952 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:56.879096 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:56.879058 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:06:57.040023 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:57.039994 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:57.946876 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:57.946840 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:58.714478 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:58.714441 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:59.468265 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:06:59.468235 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:00.129149 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:00.129114 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:00.340718 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:07:00.340683 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:07:00.877371 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:00.877330 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:01.919989 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:01.919954 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:02.557010 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:02.556977 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:03.229555 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:03.229517 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:04.016655 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:04.016624 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:04.710190 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:04.710143 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:05.433189 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:05.433152 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:05.777653 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:05.777575 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:06.124194 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:06.124165 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:06.938873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:06.938835 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:07.723792 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:07.723760 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:08.424066 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:08.424022 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:09.138198 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:09.138156 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:09.337922 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:09.337889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:09.932337 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:09.932304 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:10.672595 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:10.672569 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:11.424515 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:11.424480 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:11.739743 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:11.739652 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:07:12.472491 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:12.467104 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:13.396190 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:13.396156 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:14.039163 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:14.039130 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:14.723667 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:14.723631 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:15.425491 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:15.425445 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:16.123815 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:16.123750 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:16.837040 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:16.836984 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:17.541924 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:17.541889 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:18.212327 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:18.212298 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:18.918993 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:18.918957 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:19.646988 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:19.646787 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:20.356101 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:20.356059 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:21.030800 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:21.030765 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:21.733310 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:21.733275 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:22.423922 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:22.423880 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:23.239768 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:23.239268 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:23.489662 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:23.489630 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:24.045242 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:24.045208 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:24.720452 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:24.720422 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:25.505195 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:25.505160 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:26.217117 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:26.217091 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:26.919515 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:26.919484 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:27.116748 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:27.116720 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:27.636813 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:27.636780 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:28.348176 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:28.348143 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:29.138956 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:29.138927 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:29.837097 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:29.837060 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:30.343221 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:07:30.343185 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:07:30.554196 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:30.554152 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:31.267079 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:31.267048 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:31.943776 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:31.943743 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:32.757605 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:32.757552 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:33.534220 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:33.534186 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:34.231975 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:34.231932 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:34.901290 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:34.901258 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:35.039920 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:35.039888 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:35.846528 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:35.846496 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:07:35.919453 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:35.919422 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:36.751174 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:36.751143 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:37.578890 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:37.578856 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:38.324867 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:38.324826 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:39.087840 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:39.087809 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:39.798858 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:39.798823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:40.011919 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:40.011856 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:40.632023 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:40.631984 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:41.322767 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:41.322736 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:42.056039 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:42.055934 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:42.726566 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:42.726532 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:43.437826 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:43.437785 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:44.229877 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:44.229845 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:44.939987 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:44.939954 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:45.220743 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:45.220667 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:45.356925 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:45.356859 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:45.619993 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:45.619965 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:46.188784 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:46.188752 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:46.323330 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:46.323302 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:46.489518 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:46.489420 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:46.824228 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:46.824202 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:47.921743 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:47.921710 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:07:52.630255 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:52.630223 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:52.770013 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:52.769973 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:52.983703 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:07:52.983621 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:00.340599 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:08:00.340566 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:08:05.469163 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:08:05.469133 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:08:09.394643 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:08:09.394609 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:10.152949 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:08:10.152918 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:08:16.831705 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:08:16.831668 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:17.613539 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:08:17.613501 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:23.234548 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:08:23.234510 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:30.340579 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:08:30.340543 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:09:00.340500 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:09:00.340449 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:09:09.451510 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:09.451453 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:20.846227 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:20.846189 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:21.075310 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:21.075257 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:30.342500 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:09:30.342438 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:09:35.865039 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:35.864994 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:09:43.892142 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:43.892108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:44.030159 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:44.030126 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:51.059026 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:51.058992 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:09:55.012793 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:09:55.012763 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:10:00.342854 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:10:00.342808 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:10:09.534690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:09.534655 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:09.940105 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:09.940066 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:10:10.621825 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:10.621791 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:11.076907 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:11.076878 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:11.288172 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:11.288140 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:14.548156 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:14.548122 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:14.752585 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:14.752551 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:15.740487 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:15.740404 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:15.885997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:15.885960 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:16.631682 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:16.631638 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:17.592897 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:17.592866 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:17.930003 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:17.929926 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:10:18.436209 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:18.435449 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:19.124048 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:19.124019 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:19.827265 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:19.827181 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:20.547853 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:20.547822 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:21.286516 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:21.286454 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:22.017249 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:22.017161 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:22.771760 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:22.771732 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:23.414702 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:23.414661 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:24.209694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:24.209662 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:24.951889 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:24.951838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:25.699362 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:25.699328 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:26.326836 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:26.326798 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:27.045629 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:27.045576 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:27.748519 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:27.748482 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:28.436547 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:28.436505 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:29.136001 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:29.135973 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:29.402765 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:29.402684 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:29.955621 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:29.955592 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:30.166769 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:30.166734 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:30.331015 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:30.330985 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:30.341042 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:10:30.341013 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:10:30.886430 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:30.886403 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:31.582698 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:31.582659 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:31.923796 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:31.923714 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:32.699783 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:32.699745 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:33.348560 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:33.348532 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:10:33.667772 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:33.667696 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:34.640031 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:34.639900 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:35.321689 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:35.321652 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:36.074821 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:36.074787 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:36.943338 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:36.943249 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:37.749802 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:37.746188 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:38.450430 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:38.450391 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:39.218881 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:39.218849 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:39.921065 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:39.921035 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:40.628805 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:40.628697 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:41.346493 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:41.346344 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:42.021493 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:42.021441 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:42.841208 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:42.841159 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:43.658291 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:43.658261 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:44.360261 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:44.360215 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:44.819310 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:44.819220 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:45.036915 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:45.036872 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:45.743440 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:45.743348 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:46.477174 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:46.477133 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:47.188499 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:47.188352 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:47.898498 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:47.898432 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:48.617845 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:48.617813 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:48.862811 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:48.862780 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:10:49.361118 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:49.361081 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:50.110724 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:50.110692 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:50.966551 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:50.966518 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:51.722238 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:51.722192 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:52.445738 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:52.445704 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:53.125694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:53.125659 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:53.826411 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:53.826367 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:54.527704 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:54.527670 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:55.242635 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:55.242604 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:56.102145 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:56.102113 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:56.724440 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:56.724405 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:57.425064 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:57.425033 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:58.252093 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:58.252040 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:59.030853 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:59.030823 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:59.742967 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:10:59.742933 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:00.342018 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:11:00.341979 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:11:00.423194 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:00.423164 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:01.130295 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:01.130265 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:01.930115 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:01.930055 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:02.634290 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:02.634254 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:03.537420 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:03.537358 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:04.266367 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:04.266319 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:04.927001 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:04.926964 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:05.624449 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:05.624416 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:06.319062 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:06.319020 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:07.040689 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:07.037164 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:07.722693 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:07.722640 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:08.419614 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:08.419583 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:09.148550 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:09.148513 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:09.565844 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:09.565742 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:09.820953 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:09.820918 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:10.551454 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:10.551417 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:11.348259 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:11.348211 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:11.985430 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:11.985394 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:12.125721 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:12.125685 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:12.362668 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:12.362639 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:11:13.029881 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:13.029845 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:13.809956 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:13.809897 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:14.692314 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:14.692281 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:15.472178 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:15.472147 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:16.057621 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:16.057585 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:16.200358 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:16.200324 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:17.002113 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:17.002067 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:17.148706 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:17.148670 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:17.629896 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:17.629856 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:18.351928 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:18.351891 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:19.021262 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:19.021233 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:19.749491 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:19.747373 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:20.551211 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:20.551051 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:21.226041 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:21.225994 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:21.939663 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:21.939614 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:22.331285 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:22.331258 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:22.476946 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:22.476897 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:22.642400 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:22.642324 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:23.237193 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:23.237161 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:23.465964 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:23.465924 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:23.602134 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:23.602097 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:23.992670 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:23.992586 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:24.883516 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:24.883481 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:11:29.897089 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:29.897058 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:30.031803 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:30.031774 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:30.245761 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:30.245669 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:30.343775 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:11:30.343739 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:11:34.803458 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:34.803421 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:36.149622 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:36.149586 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:11:39.149674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:39.149629 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:11:39.150114 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:39.149709 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:11:50.156332 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:50.156246 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:51.767570 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:51.767535 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:11:54.457821 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:54.457787 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:55.193491 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:11:55.189944 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:00.341662 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:12:00.341624 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:12:01.284623 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:01.284592 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:08.057363 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:08.057324 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:12:09.640062 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:09.640001 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:10.229942 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:10.229871 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:11.193487 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:11.189599 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hp9kr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:12:11.193487 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:11.189698 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hp9kr" podUID="e5f06403-3135-46e2-bf0e-7a08611c8d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:12:18.372369 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:18.372316 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:19.825348 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:19.825314 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:12:20.630203 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:20.630109 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:24.578232 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:24.578191 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:26.494302 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:26.494261 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:12:30.342635 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:12:30.342596 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:12:34.963392 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:34.963356 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:12:39.152677 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:39.152597 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:12:39.153220 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:39.152714 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:12:39.407020 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:39.406929 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:12:56.152558 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:12:56.152525 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:13:00.340443 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:13:00.340407 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:13:04.180348 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:04.180309 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:04.655426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:04.655384 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:04.839701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:04.839668 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:07.712137 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:07.712106 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:07.847929 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:07.847903 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:08.974069 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:08.974039 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:09.114189 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:09.114152 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:09.248990 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:09.248909 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:09.380974 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:09.380940 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:09.678299 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:09.678256 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:09.926723 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:09.926692 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:10.306861 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:10.306799 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:10.850871 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:10.850836 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:11.025928 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:11.025886 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:13:11.615774 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:11.615747 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:12.353118 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:12.353056 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:13.070238 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:13.070203 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:13.154691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:13.154658 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-kstmg"] Oct 31 13:13:13.154917 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:13.154887 6727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" podUID="8cc1277f-a661-4d8c-8402-4b90554f0ba9" containerName="maas-api" containerID="cri-o://21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf" gracePeriod=30 Oct 31 13:13:13.944475 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:13.944430 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:14.400158 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.400134 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 13:13:14.437186 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.437160 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc1277f-a661-4d8c-8402-4b90554f0ba9-kube-api-access-fl2bs\" (UniqueName: \"kubernetes.io/projected/8cc1277f-a661-4d8c-8402-4b90554f0ba9-kube-api-access-fl2bs\") pod \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " Oct 31 13:13:14.437356 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.437226 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc1277f-a661-4d8c-8402-4b90554f0ba9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc1277f-a661-4d8c-8402-4b90554f0ba9-openshift-service-ca\") pod \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " Oct 31 13:13:14.437356 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.437258 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-maas-api-tls\") pod \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " Oct 31 13:13:14.437356 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.437317 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-metrics-certs\") pod \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\" (UID: \"8cc1277f-a661-4d8c-8402-4b90554f0ba9\") " Oct 31 13:13:14.437702 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.437679 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc1277f-a661-4d8c-8402-4b90554f0ba9-openshift-service-ca" pod "8cc1277f-a661-4d8c-8402-4b90554f0ba9" (UID: "8cc1277f-a661-4d8c-8402-4b90554f0ba9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:14.437819 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.437803 6727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc1277f-a661-4d8c-8402-4b90554f0ba9-openshift-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:14.439444 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.439418 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-maas-api-tls" pod "8cc1277f-a661-4d8c-8402-4b90554f0ba9" (UID: "8cc1277f-a661-4d8c-8402-4b90554f0ba9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:14.439579 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.439418 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc1277f-a661-4d8c-8402-4b90554f0ba9-kube-api-access-fl2bs" pod "8cc1277f-a661-4d8c-8402-4b90554f0ba9" (UID: "8cc1277f-a661-4d8c-8402-4b90554f0ba9"). InnerVolumeSpecName "kube-api-access-fl2bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:14.439579 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.439533 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-metrics-certs" pod "8cc1277f-a661-4d8c-8402-4b90554f0ba9" (UID: "8cc1277f-a661-4d8c-8402-4b90554f0ba9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:14.539323 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.539265 6727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-metrics-certs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:14.539323 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.539289 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl2bs\" (UniqueName: \"kubernetes.io/projected/8cc1277f-a661-4d8c-8402-4b90554f0ba9-kube-api-access-fl2bs\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:14.539323 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.539300 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc1277f-a661-4d8c-8402-4b90554f0ba9-maas-api-tls\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:14.774318 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:14.774262 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:15.250342 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.250305 6727 generic.go:356] "Generic (PLEG): container finished" podID="8cc1277f-a661-4d8c-8402-4b90554f0ba9" containerID="21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf" exitCode=0 Oct 31 13:13:15.250524 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.250352 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" Oct 31 13:13:15.250524 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.250391 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" event={"ID":"8cc1277f-a661-4d8c-8402-4b90554f0ba9","Type":"ContainerDied","Data":"21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf"} Oct 31 13:13:15.250524 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.250442 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-kstmg" event={"ID":"8cc1277f-a661-4d8c-8402-4b90554f0ba9","Type":"ContainerDied","Data":"eb6f21d3398d130e79f1d08c9cf5974b454c6fc8068531ad654047f74917f45f"} Oct 31 13:13:15.250524 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.250488 6727 scope.go:122] "RemoveContainer" containerID="21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf" Oct 31 13:13:15.260331 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.260314 6727 scope.go:122] "RemoveContainer" containerID="21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf" Oct 31 13:13:15.260614 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:13:15.260596 6727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf\": container with ID starting with 21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf not found: ID does not exist" containerID="21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf" Oct 31 13:13:15.260677 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.260622 6727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf"} err="failed to get container status \"21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf\": rpc error: code = NotFound desc = could not find container \"21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf\": container with ID starting with 21d7abeb98523ae72b35c464e80c3ee7a5eb1abb1591b23b7a68624fb616d5bf not found: ID does not exist" Oct 31 13:13:15.283001 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.282980 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-kstmg"] Oct 31 13:13:15.289397 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.289375 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-kstmg"] Oct 31 13:13:15.298414 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.298393 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc1277f-a661-4d8c-8402-4b90554f0ba9" path="/var/lib/kubelet/pods/8cc1277f-a661-4d8c-8402-4b90554f0ba9/volumes" Oct 31 13:13:15.543783 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:15.543727 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:16.244989 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:16.244954 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:17.147790 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:17.147752 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:17.993003 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:17.992967 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:18.823326 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:18.823267 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:19.585379 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:19.585298 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:20.300874 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:20.300847 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:21.028427 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:21.028391 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:21.850103 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:21.846367 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:22.689457 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:22.689425 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:23.346973 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.346938 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q"] Oct 31 13:13:23.347378 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347364 6727 cpu_manager.go:414] "containerMap: removing container" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" Oct 31 13:13:23.347426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347378 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" Oct 31 13:13:23.347426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347401 6727 cpu_manager.go:414] "containerMap: removing container" podUID="8cc1277f-a661-4d8c-8402-4b90554f0ba9" containerName="maas-api" Oct 31 13:13:23.347426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347406 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc1277f-a661-4d8c-8402-4b90554f0ba9" containerName="maas-api" Oct 31 13:13:23.347426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347416 6727 cpu_manager.go:414] "containerMap: removing container" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" Oct 31 13:13:23.347426 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347422 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dd643e-ae00-46a6-9d8a-a7f8aaf636aa" containerName="istio-proxy" Oct 31 13:13:23.347638 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347502 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8b2d25-b0ad-4591-b480-c02c4e0fe43d" containerName="istio-proxy" Oct 31 13:13:23.347638 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.347514 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc1277f-a661-4d8c-8402-4b90554f0ba9" containerName="maas-api" Oct 31 13:13:23.350587 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.350569 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.355266 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.355243 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:13:23.355406 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.355288 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:13:23.355406 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.355310 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\"" Oct 31 13:13:23.355406 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.355319 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:13:23.363499 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.363457 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q"] Oct 31 13:13:23.422142 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.422117 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.422244 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.422155 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.422303 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.422264 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6qg\" (UniqueName: \"kubernetes.io/projected/2b715ec7-9644-430f-9f84-349b9abb9563-kube-api-access-tf6qg\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.446035 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.446001 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:23.523893 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.523860 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.524067 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.523912 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.524067 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.523997 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6qg\" (UniqueName: \"kubernetes.io/projected/2b715ec7-9644-430f-9f84-349b9abb9563-kube-api-access-tf6qg\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.524684 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.524660 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.526265 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.526244 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.535873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.535850 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6qg\" (UniqueName: \"kubernetes.io/projected/2b715ec7-9644-430f-9f84-349b9abb9563-kube-api-access-tf6qg\") pod \"maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.664199 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.664125 6727 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" secret="" err="secret \"maas-api-cleanup-dockercfg-7wsmb\" not found" Oct 31 13:13:23.664199 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.664170 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:23.942777 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.942681 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q"] Oct 31 13:13:23.945198 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:13:23.945169 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b715ec7_9644_430f_9f84_349b9abb9563.slice/crio-2e52aeb3663c2c2d110c9e2d3b1c1645b17deda1656f889b887128d2364ca20f WatchSource:0}: Error finding container 2e52aeb3663c2c2d110c9e2d3b1c1645b17deda1656f889b887128d2364ca20f: Status 404 returned error can't find the container with id 2e52aeb3663c2c2d110c9e2d3b1c1645b17deda1656f889b887128d2364ca20f Oct 31 13:13:23.954791 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:23.954758 6727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:13:24.311237 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:24.311172 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" event={"ID":"2b715ec7-9644-430f-9f84-349b9abb9563","Type":"ContainerStarted","Data":"2e52aeb3663c2c2d110c9e2d3b1c1645b17deda1656f889b887128d2364ca20f"} Oct 31 13:13:24.354738 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:24.354713 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:25.258187 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:25.258150 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:26.057393 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:26.057360 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:26.846378 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:26.846238 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:27.325630 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:27.325602 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" event={"ID":"2b715ec7-9644-430f-9f84-349b9abb9563","Type":"ContainerStarted","Data":"7e5d3ff3d3ec06a86639c5cc615341d99dd397eb4433f04c62fbbacdd66dbe54"} Oct 31 13:13:27.326037 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:27.325736 6727 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" secret="" err="secret \"maas-api-cleanup-dockercfg-7wsmb\" not found" Oct 31 13:13:27.343540 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:27.343494 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" podStartSLOduration=1.662155745 podStartE2EDuration="4.343482196s" totalImagesPullingTime="2.681326451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:27.342258898 +0000 UTC m=+2718.157273291" watchObservedRunningTime="2026-10-31 13:13:27.343482196 +0000 UTC m=+2718.158496578" Oct 31 13:13:27.525055 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:27.525014 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:28.227047 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:28.227014 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:28.333256 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:28.333226 6727 generic.go:356] "Generic (PLEG): container finished" podID="2b715ec7-9644-430f-9f84-349b9abb9563" containerID="7e5d3ff3d3ec06a86639c5cc615341d99dd397eb4433f04c62fbbacdd66dbe54" exitCode=0 Oct 31 13:13:28.333685 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:28.333314 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" event={"ID":"2b715ec7-9644-430f-9f84-349b9abb9563","Type":"ContainerDied","Data":"7e5d3ff3d3ec06a86639c5cc615341d99dd397eb4433f04c62fbbacdd66dbe54"} Oct 31 13:13:28.938106 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:28.938006 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:29.470182 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.470161 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:29.587289 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.587259 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-xks-ca\") pod \"2b715ec7-9644-430f-9f84-349b9abb9563\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " Oct 31 13:13:29.587484 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.587307 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-openshift-ca\") pod \"2b715ec7-9644-430f-9f84-349b9abb9563\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " Oct 31 13:13:29.587484 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.587332 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b715ec7-9644-430f-9f84-349b9abb9563-kube-api-access-tf6qg\" (UniqueName: \"kubernetes.io/projected/2b715ec7-9644-430f-9f84-349b9abb9563-kube-api-access-tf6qg\") pod \"2b715ec7-9644-430f-9f84-349b9abb9563\" (UID: \"2b715ec7-9644-430f-9f84-349b9abb9563\") " Oct 31 13:13:29.587674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.587652 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-openshift-ca" pod "2b715ec7-9644-430f-9f84-349b9abb9563" (UID: "2b715ec7-9644-430f-9f84-349b9abb9563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:29.589253 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.589189 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-xks-ca" pod "2b715ec7-9644-430f-9f84-349b9abb9563" (UID: "2b715ec7-9644-430f-9f84-349b9abb9563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:29.589559 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.589540 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b715ec7-9644-430f-9f84-349b9abb9563-kube-api-access-tf6qg" pod "2b715ec7-9644-430f-9f84-349b9abb9563" (UID: "2b715ec7-9644-430f-9f84-349b9abb9563"). InnerVolumeSpecName "kube-api-access-tf6qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:29.624882 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.624859 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:29.689185 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.689147 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-xks-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.689185 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.689179 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b715ec7-9644-430f-9f84-349b9abb9563-maas-api-openshift-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.689185 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:29.689190 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6qg\" (UniqueName: \"kubernetes.io/projected/2b715ec7-9644-430f-9f84-349b9abb9563-kube-api-access-tf6qg\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:30.329072 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:30.329044 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:30.358602 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:30.358557 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" Oct 31 13:13:30.359623 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:30.358628 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q" event={"ID":"2b715ec7-9644-430f-9f84-349b9abb9563","Type":"ContainerDied","Data":"2e52aeb3663c2c2d110c9e2d3b1c1645b17deda1656f889b887128d2364ca20f"} Oct 31 13:13:30.359775 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:30.359650 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e52aeb3663c2c2d110c9e2d3b1c1645b17deda1656f889b887128d2364ca20f" Oct 31 13:13:30.361171 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:13:30.361139 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:13:31.027416 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:31.027387 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:31.753649 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:31.753615 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:32.423856 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:32.423825 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:33.156832 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:33.156802 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:33.600862 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:33.600831 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:33.861362 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:33.861291 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:34.012484 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:34.009667 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:34.589728 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:34.589699 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q"] Oct 31 13:13:34.598167 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:34.598123 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf78971-j4g7q"] Oct 31 13:13:34.620958 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:34.620934 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:34.756861 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:34.756813 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:35.299099 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:35.299071 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b715ec7-9644-430f-9f84-349b9abb9563" path="/var/lib/kubelet/pods/2b715ec7-9644-430f-9f84-349b9abb9563/volumes" Oct 31 13:13:36.057425 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:36.057388 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:13:39.697577 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:39.697542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:40.868288 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:40.868252 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:41.024614 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:41.024557 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:41.287859 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:41.287782 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:41.505826 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:41.505795 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:44.738708 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:44.738674 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:44.870522 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:44.870496 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:46.101429 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:46.101398 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:46.276045 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:46.276006 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:46.952652 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:46.952612 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:47.933579 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:47.933539 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:48.162903 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:48.162868 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:13:48.697764 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:48.697706 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:49.342828 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:49.342758 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:50.048338 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:50.046691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:50.769879 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:50.769844 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:51.438017 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:51.437629 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:52.126799 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:52.126756 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:52.822489 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:52.822437 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:53.509800 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:53.509757 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:54.266816 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:54.266774 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:55.137854 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:55.137820 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:55.833224 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:55.833193 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:56.561113 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:56.561027 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:57.233482 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:57.233424 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:57.953626 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:57.953596 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:58.630609 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:58.630561 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:59.447323 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:59.447286 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:59.683665 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:13:59.683631 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:00.149177 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:00.149108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:00.341175 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:14:00.341134 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:14:00.861855 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:00.861819 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:01.818594 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:01.818565 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:02.524473 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:02.524442 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:03.215038 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:03.215009 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:03.982315 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:03.982274 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:04.637225 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:04.637184 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:05.314308 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:05.314280 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:06.036558 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:06.036511 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:06.733505 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:06.733026 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:07.443125 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:07.443088 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:08.121014 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:08.120972 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:08.926634 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:08.926601 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:09.676921 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:09.676874 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:09.808053 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:09.808024 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:10.129565 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:10.129535 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:10.428732 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:10.428647 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:11.242674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:11.242641 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:11.923951 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:11.923908 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:12.653798 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:12.653761 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:13.423258 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:13.423223 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:14.152214 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:14.152176 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:14.829811 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:14.829777 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:15.542722 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:15.542686 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:16.251813 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:16.251764 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:17.021340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:17.021309 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:17.842803 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:17.842771 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:18.558449 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:18.552254 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:19.315182 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:19.315133 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:20.026661 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:20.026622 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:20.736371 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:20.736335 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:21.633610 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:21.633573 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:22.441630 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:22.441107 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:23.126031 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:23.125997 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:23.841140 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:23.841108 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:24.531942 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:24.531909 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:25.233354 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:25.233323 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:25.923367 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:25.923334 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:26.641131 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:26.641048 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:27.355112 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:27.355077 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:28.019133 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:28.019102 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:28.839253 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:28.839222 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:29.545141 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:29.540652 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:30.299936 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:30.299908 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:30.344929 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:14:30.344830 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:14:31.027486 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:31.027420 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:31.717878 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:31.717846 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:32.424718 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:32.424682 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:33.212058 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:33.212030 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:33.971314 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:33.971280 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:34.643710 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:34.643608 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:35.412675 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:35.412643 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:36.283124 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:36.283094 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:37.016725 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:37.016665 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:37.883013 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:37.882977 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:38.521035 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:38.521000 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:39.219333 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:39.219300 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:39.914976 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:39.914942 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:40.618266 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:40.618233 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:41.333872 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:41.333838 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:42.033780 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:42.033734 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:42.729437 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:42.729402 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:43.438601 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:43.438552 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:44.124800 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:44.124769 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:44.870362 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:44.870303 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:45.535427 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:45.528791 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:46.240967 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:46.240933 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:46.915630 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:46.915590 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:47.653013 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:47.652977 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:48.348581 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:48.348542 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:49.026348 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:49.026316 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:49.737951 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:49.736518 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:50.559791 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:50.559763 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:51.250447 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:51.250412 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:52.039483 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:52.036955 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:52.729155 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:52.729109 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:53.621643 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:53.621612 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:54.351691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:54.351653 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:55.042852 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:55.032240 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:55.731521 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:55.731478 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:56.416346 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:56.416314 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:57.116084 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:57.116048 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:57.925291 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:57.925256 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:58.618365 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:58.618333 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:59.321532 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:14:59.321498 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:00.023395 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.023367 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:00.143360 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.143328 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6"] Oct 31 13:15:00.143786 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.143773 6727 cpu_manager.go:414] "containerMap: removing container" podUID="2b715ec7-9644-430f-9f84-349b9abb9563" containerName="revoke-keys" Oct 31 13:15:00.143841 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.143787 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b715ec7-9644-430f-9f84-349b9abb9563" containerName="revoke-keys" Oct 31 13:15:00.143879 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.143857 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b715ec7-9644-430f-9f84-349b9abb9563" containerName="revoke-keys" Oct 31 13:15:00.146848 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.146825 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r"] Oct 31 13:15:00.146981 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.146919 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.150315 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.150297 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.154246 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.154169 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:15:00.154246 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.154208 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:15:00.155568 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.155547 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x5v7\"" Oct 31 13:15:00.156160 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.156141 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6"] Oct 31 13:15:00.158842 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.158820 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r"] Oct 31 13:15:00.340792 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:00.340764 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:15:00.342176 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.342156 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c85787-503d-4064-9118-6f0393498825-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890875-vwv5r\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.342276 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.342262 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vzw\" (UniqueName: \"kubernetes.io/projected/f5c85787-503d-4064-9118-6f0393498825-kube-api-access-79vzw\") pod \"maas-api-key-cleanup-29890875-vwv5r\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.342343 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.342325 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3114d-dff3-46ca-ba25-e051289b7640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.342381 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.342352 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pll46\" (UniqueName: \"kubernetes.io/projected/e9d3114d-dff3-46ca-ba25-e051289b7640-kube-api-access-pll46\") pod \"maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.443513 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.443490 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vzw\" (UniqueName: \"kubernetes.io/projected/f5c85787-503d-4064-9118-6f0393498825-kube-api-access-79vzw\") pod \"maas-api-key-cleanup-29890875-vwv5r\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.443651 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.443534 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3114d-dff3-46ca-ba25-e051289b7640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.443651 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.443555 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pll46\" (UniqueName: \"kubernetes.io/projected/e9d3114d-dff3-46ca-ba25-e051289b7640-kube-api-access-pll46\") pod \"maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.443825 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.443803 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c85787-503d-4064-9118-6f0393498825-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890875-vwv5r\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.444268 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.444250 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3114d-dff3-46ca-ba25-e051289b7640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.444438 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.444421 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c85787-503d-4064-9118-6f0393498825-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890875-vwv5r\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.455375 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.455348 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vzw\" (UniqueName: \"kubernetes.io/projected/f5c85787-503d-4064-9118-6f0393498825-kube-api-access-79vzw\") pod \"maas-api-key-cleanup-29890875-vwv5r\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.455521 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.455435 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pll46\" (UniqueName: \"kubernetes.io/projected/e9d3114d-dff3-46ca-ba25-e051289b7640-kube-api-access-pll46\") pod \"maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.463396 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.463177 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:00.470493 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.470472 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:00.755540 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.754715 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r"] Oct 31 13:15:00.755540 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.754822 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6"] Oct 31 13:15:00.803115 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.803087 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" event={"ID":"f5c85787-503d-4064-9118-6f0393498825","Type":"ContainerStarted","Data":"57d9cffacdaabfae3860534aa9dd97c7c8ad1faec0ec216cf530aff2d00b3bc5"} Oct 31 13:15:00.804249 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.804227 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" event={"ID":"e9d3114d-dff3-46ca-ba25-e051289b7640","Type":"ContainerStarted","Data":"ce71c9f2790d5161b17f99562a6b758de6fe7be874b3389c6562d6ef57b6d693"} Oct 31 13:15:00.856579 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:00.850848 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:01.541452 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:01.541415 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:01.811044 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:01.811006 6727 generic.go:356] "Generic (PLEG): container finished" podID="f5c85787-503d-4064-9118-6f0393498825" containerID="c0a5bb3a93c0c3d2b562226d1c45b9ee6105a88e9e2072069547bdb3ae6d4559" exitCode=60 Oct 31 13:15:01.811191 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:01.811083 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" event={"ID":"f5c85787-503d-4064-9118-6f0393498825","Type":"ContainerDied","Data":"c0a5bb3a93c0c3d2b562226d1c45b9ee6105a88e9e2072069547bdb3ae6d4559"} Oct 31 13:15:01.811368 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:01.811353 6727 scope.go:122] "RemoveContainer" containerID="c0a5bb3a93c0c3d2b562226d1c45b9ee6105a88e9e2072069547bdb3ae6d4559" Oct 31 13:15:01.812975 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:01.812954 6727 generic.go:356] "Generic (PLEG): container finished" podID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerID="56268746b010ad284cc72b6e87919e512993483b6c88082f2de63eb23ce21ce5" exitCode=60 Oct 31 13:15:01.813058 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:01.813001 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" event={"ID":"e9d3114d-dff3-46ca-ba25-e051289b7640","Type":"ContainerDied","Data":"56268746b010ad284cc72b6e87919e512993483b6c88082f2de63eb23ce21ce5"} Oct 31 13:15:01.813270 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:01.813254 6727 scope.go:122] "RemoveContainer" containerID="56268746b010ad284cc72b6e87919e512993483b6c88082f2de63eb23ce21ce5" Oct 31 13:15:02.224933 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.224906 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:02.820330 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.820300 6727 generic.go:356] "Generic (PLEG): container finished" podID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerID="1aebb7403344eb4e09e306f901dbb468b466b07a8302577f3a00d1ed326ef546" exitCode=60 Oct 31 13:15:02.820722 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.820379 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" event={"ID":"e9d3114d-dff3-46ca-ba25-e051289b7640","Type":"ContainerDied","Data":"1aebb7403344eb4e09e306f901dbb468b466b07a8302577f3a00d1ed326ef546"} Oct 31 13:15:02.820722 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.820431 6727 scope.go:122] "RemoveContainer" containerID="56268746b010ad284cc72b6e87919e512993483b6c88082f2de63eb23ce21ce5" Oct 31 13:15:02.820722 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.820618 6727 scope.go:122] "RemoveContainer" containerID="1aebb7403344eb4e09e306f901dbb468b466b07a8302577f3a00d1ed326ef546" Oct 31 13:15:02.820866 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:02.820824 6727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6_odh-ai-gateway-infra(e9d3114d-dff3-46ca-ba25-e051289b7640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" Oct 31 13:15:02.822427 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.822409 6727 generic.go:356] "Generic (PLEG): container finished" podID="f5c85787-503d-4064-9118-6f0393498825" containerID="f58082ecccde9237a61bc8d236ec536e7a87da9654098b06b2f7c30fcbf21f0b" exitCode=60 Oct 31 13:15:02.822545 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.822486 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" event={"ID":"f5c85787-503d-4064-9118-6f0393498825","Type":"ContainerDied","Data":"f58082ecccde9237a61bc8d236ec536e7a87da9654098b06b2f7c30fcbf21f0b"} Oct 31 13:15:02.822673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.822660 6727 scope.go:122] "RemoveContainer" containerID="f58082ecccde9237a61bc8d236ec536e7a87da9654098b06b2f7c30fcbf21f0b" Oct 31 13:15:02.822843 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:02.822827 6727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890875-vwv5r_odh-ai-gateway-infra(f5c85787-503d-4064-9118-6f0393498825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" podUID="f5c85787-503d-4064-9118-6f0393498825" Oct 31 13:15:02.832365 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:02.832347 6727 scope.go:122] "RemoveContainer" containerID="c0a5bb3a93c0c3d2b562226d1c45b9ee6105a88e9e2072069547bdb3ae6d4559" Oct 31 13:15:03.018309 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:03.018229 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:03.773131 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:03.773097 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:03.829119 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:03.829092 6727 scope.go:122] "RemoveContainer" containerID="f58082ecccde9237a61bc8d236ec536e7a87da9654098b06b2f7c30fcbf21f0b" Oct 31 13:15:03.829596 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:03.829366 6727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890875-vwv5r_odh-ai-gateway-infra(f5c85787-503d-4064-9118-6f0393498825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" podUID="f5c85787-503d-4064-9118-6f0393498825" Oct 31 13:15:03.830676 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:03.830658 6727 scope.go:122] "RemoveContainer" containerID="1aebb7403344eb4e09e306f901dbb468b466b07a8302577f3a00d1ed326ef546" Oct 31 13:15:03.830876 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:03.830858 6727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6_odh-ai-gateway-infra(e9d3114d-dff3-46ca-ba25-e051289b7640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" Oct 31 13:15:04.516680 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:04.516647 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:05.219270 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:05.219237 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:05.917359 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:05.917328 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:06.618852 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:06.618821 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:07.338120 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:07.338060 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:08.027535 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:08.027501 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:08.739626 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:08.739582 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:09.546382 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:09.546337 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:09.807018 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:09.806906 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:10.182197 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:10.182148 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:10.328852 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:10.328808 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:10.933681 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:10.933645 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:11.658776 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:11.658732 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:12.326533 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:12.326499 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:13.021652 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:13.021617 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:13.293678 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:13.293605 6727 scope.go:122] "RemoveContainer" containerID="f58082ecccde9237a61bc8d236ec536e7a87da9654098b06b2f7c30fcbf21f0b" Oct 31 13:15:13.729481 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:13.728723 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:13.883527 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:13.883494 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" event={"ID":"f5c85787-503d-4064-9118-6f0393498825","Type":"ContainerStarted","Data":"0bbb2a3189a49147cfe7752c6ebbe5e74bda77ceb2e72e67dd5c956546b563d9"} Oct 31 13:15:13.902521 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:13.902476 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" podStartSLOduration=13.902446742 podStartE2EDuration="13.902446742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:13.90078293 +0000 UTC m=+2824.715797322" watchObservedRunningTime="2026-10-31 13:15:13.902446742 +0000 UTC m=+2824.717461167" Oct 31 13:15:14.430524 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:14.430422 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:14.890269 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:14.890234 6727 generic.go:356] "Generic (PLEG): container finished" podID="f5c85787-503d-4064-9118-6f0393498825" containerID="0bbb2a3189a49147cfe7752c6ebbe5e74bda77ceb2e72e67dd5c956546b563d9" exitCode=60 Oct 31 13:15:14.890724 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:14.890307 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" event={"ID":"f5c85787-503d-4064-9118-6f0393498825","Type":"ContainerDied","Data":"0bbb2a3189a49147cfe7752c6ebbe5e74bda77ceb2e72e67dd5c956546b563d9"} Oct 31 13:15:14.890724 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:14.890366 6727 scope.go:122] "RemoveContainer" containerID="f58082ecccde9237a61bc8d236ec536e7a87da9654098b06b2f7c30fcbf21f0b" Oct 31 13:15:14.890724 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:14.890647 6727 scope.go:122] "RemoveContainer" containerID="0bbb2a3189a49147cfe7752c6ebbe5e74bda77ceb2e72e67dd5c956546b563d9" Oct 31 13:15:14.890905 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:14.890884 6727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890875-vwv5r_odh-ai-gateway-infra(f5c85787-503d-4064-9118-6f0393498825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" podUID="f5c85787-503d-4064-9118-6f0393498825" Oct 31 13:15:14.912765 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:14.912738 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r"] Oct 31 13:15:15.118601 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:15.118574 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:15.919814 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:15.919777 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:16.041403 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.041375 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:16.097053 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.097022 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c85787-503d-4064-9118-6f0393498825-kube-api-access-79vzw\" (UniqueName: \"kubernetes.io/projected/f5c85787-503d-4064-9118-6f0393498825-kube-api-access-79vzw\") pod \"f5c85787-503d-4064-9118-6f0393498825\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " Oct 31 13:15:16.097207 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.097190 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c85787-503d-4064-9118-6f0393498825-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c85787-503d-4064-9118-6f0393498825-maas-api-service-ca\") pod \"f5c85787-503d-4064-9118-6f0393498825\" (UID: \"f5c85787-503d-4064-9118-6f0393498825\") " Oct 31 13:15:16.097585 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.097564 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c85787-503d-4064-9118-6f0393498825-maas-api-service-ca" pod "f5c85787-503d-4064-9118-6f0393498825" (UID: "f5c85787-503d-4064-9118-6f0393498825"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:16.099075 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.099045 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c85787-503d-4064-9118-6f0393498825-kube-api-access-79vzw" pod "f5c85787-503d-4064-9118-6f0393498825" (UID: "f5c85787-503d-4064-9118-6f0393498825"). InnerVolumeSpecName "kube-api-access-79vzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:16.198740 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.198687 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c85787-503d-4064-9118-6f0393498825-maas-api-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:16.198740 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.198708 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79vzw\" (UniqueName: \"kubernetes.io/projected/f5c85787-503d-4064-9118-6f0393498825-kube-api-access-79vzw\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:16.716755 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.716720 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:16.903174 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.903136 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" event={"ID":"f5c85787-503d-4064-9118-6f0393498825","Type":"ContainerDied","Data":"57d9cffacdaabfae3860534aa9dd97c7c8ad1faec0ec216cf530aff2d00b3bc5"} Oct 31 13:15:16.903358 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.903220 6727 scope.go:122] "RemoveContainer" containerID="0bbb2a3189a49147cfe7752c6ebbe5e74bda77ceb2e72e67dd5c956546b563d9" Oct 31 13:15:16.903358 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.903145 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r" Oct 31 13:15:16.924952 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.924933 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r"] Oct 31 13:15:16.932925 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:16.932903 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-vwv5r"] Oct 31 13:15:17.293069 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:17.293041 6727 scope.go:122] "RemoveContainer" containerID="1aebb7403344eb4e09e306f901dbb468b466b07a8302577f3a00d1ed326ef546" Oct 31 13:15:17.299251 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:17.299222 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c85787-503d-4064-9118-6f0393498825" path="/var/lib/kubelet/pods/f5c85787-503d-4064-9118-6f0393498825/volumes" Oct 31 13:15:17.485897 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:17.485868 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:17.911025 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:17.910939 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" event={"ID":"e9d3114d-dff3-46ca-ba25-e051289b7640","Type":"ContainerStarted","Data":"2c5365b793a0cb87ad3eaa2778d944a0d49a73ee9d38b69ede6e55a92084fbd6"} Oct 31 13:15:17.936200 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:17.936148 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" podStartSLOduration=17.936135731 podStartE2EDuration="17.936135731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:17.935179033 +0000 UTC m=+2828.750193425" watchObservedRunningTime="2026-10-31 13:15:17.936135731 +0000 UTC m=+2828.751150122" Oct 31 13:15:18.357584 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:18.357536 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:18.919256 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:18.919222 6727 generic.go:356] "Generic (PLEG): container finished" podID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerID="2c5365b793a0cb87ad3eaa2778d944a0d49a73ee9d38b69ede6e55a92084fbd6" exitCode=60 Oct 31 13:15:18.919442 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:18.919294 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" event={"ID":"e9d3114d-dff3-46ca-ba25-e051289b7640","Type":"ContainerDied","Data":"2c5365b793a0cb87ad3eaa2778d944a0d49a73ee9d38b69ede6e55a92084fbd6"} Oct 31 13:15:18.919442 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:18.919355 6727 scope.go:122] "RemoveContainer" containerID="1aebb7403344eb4e09e306f901dbb468b466b07a8302577f3a00d1ed326ef546" Oct 31 13:15:18.919750 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:18.919735 6727 scope.go:122] "RemoveContainer" containerID="2c5365b793a0cb87ad3eaa2778d944a0d49a73ee9d38b69ede6e55a92084fbd6" Oct 31 13:15:18.920043 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:18.920023 6727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6_odh-ai-gateway-infra(e9d3114d-dff3-46ca-ba25-e051289b7640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" Oct 31 13:15:18.946304 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:18.946273 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6"] Oct 31 13:15:19.250231 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:19.250125 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:20.037694 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.037559 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:20.073097 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.073074 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:20.139036 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.139007 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d3114d-dff3-46ca-ba25-e051289b7640-kube-api-access-pll46\" (UniqueName: \"kubernetes.io/projected/e9d3114d-dff3-46ca-ba25-e051289b7640-kube-api-access-pll46\") pod \"e9d3114d-dff3-46ca-ba25-e051289b7640\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " Oct 31 13:15:20.139194 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.139117 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d3114d-dff3-46ca-ba25-e051289b7640-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3114d-dff3-46ca-ba25-e051289b7640-maas-api-service-ca\") pod \"e9d3114d-dff3-46ca-ba25-e051289b7640\" (UID: \"e9d3114d-dff3-46ca-ba25-e051289b7640\") " Oct 31 13:15:20.139442 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.139418 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d3114d-dff3-46ca-ba25-e051289b7640-maas-api-service-ca" pod "e9d3114d-dff3-46ca-ba25-e051289b7640" (UID: "e9d3114d-dff3-46ca-ba25-e051289b7640"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:20.141074 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.141052 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d3114d-dff3-46ca-ba25-e051289b7640-kube-api-access-pll46" pod "e9d3114d-dff3-46ca-ba25-e051289b7640" (UID: "e9d3114d-dff3-46ca-ba25-e051289b7640"). InnerVolumeSpecName "kube-api-access-pll46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:20.240714 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.240648 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3114d-dff3-46ca-ba25-e051289b7640-maas-api-service-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:20.240714 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.240675 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pll46\" (UniqueName: \"kubernetes.io/projected/e9d3114d-dff3-46ca-ba25-e051289b7640-kube-api-access-pll46\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:20.770942 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.770907 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:20.933776 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.933749 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" Oct 31 13:15:20.933962 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.933770 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6" event={"ID":"e9d3114d-dff3-46ca-ba25-e051289b7640","Type":"ContainerDied","Data":"ce71c9f2790d5161b17f99562a6b758de6fe7be874b3389c6562d6ef57b6d693"} Oct 31 13:15:20.933962 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.933857 6727 scope.go:122] "RemoveContainer" containerID="2c5365b793a0cb87ad3eaa2778d944a0d49a73ee9d38b69ede6e55a92084fbd6" Oct 31 13:15:20.966340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.966317 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6"] Oct 31 13:15:20.969400 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:20.969373 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-198ea4-29890875-4wck6"] Oct 31 13:15:21.298805 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:21.298776 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" path="/var/lib/kubelet/pods/e9d3114d-dff3-46ca-ba25-e051289b7640/volumes" Oct 31 13:15:21.775647 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:21.775613 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:22.416131 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416095 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88"] Oct 31 13:15:22.416604 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416586 6727 cpu_manager.go:414] "containerMap: removing container" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:22.416604 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416605 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:22.416691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416620 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:22.416691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416626 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:22.416691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416643 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:22.416691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416650 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:22.416691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416659 6727 cpu_manager.go:414] "containerMap: removing container" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:22.416691 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416664 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:22.416879 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416752 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:22.416879 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416760 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:22.416879 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416775 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:22.416879 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.416782 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:22.421606 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.421576 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.447040 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.447010 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88"] Oct 31 13:15:22.471389 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.471355 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:15:22.471389 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.471356 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:15:22.479882 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.479862 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:15:22.480011 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.479987 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x5v7\"" Oct 31 13:15:22.565177 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.565152 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.565290 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.565212 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2n7d\" (UniqueName: \"kubernetes.io/projected/5bb125a0-c715-407f-900f-5ef667b92c05-kube-api-access-l2n7d\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.565340 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.565302 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.653586 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.653544 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:22.666407 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.666346 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2n7d\" (UniqueName: \"kubernetes.io/projected/5bb125a0-c715-407f-900f-5ef667b92c05-kube-api-access-l2n7d\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.666603 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.666399 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.666603 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.666545 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.667162 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.667142 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.668784 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.668763 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.682440 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.682414 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2n7d\" (UniqueName: \"kubernetes.io/projected/5bb125a0-c715-407f-900f-5ef667b92c05-kube-api-access-l2n7d\") pod \"maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:22.734969 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:22.734948 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:23.008632 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:23.008605 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88"] Oct 31 13:15:23.010662 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:15:23.010634 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb125a0_c715_407f_900f_5ef667b92c05.slice/crio-10ae15e0c5332779a3d8ca5a2e742705c4434664395e4787fc4b299ad9444477 WatchSource:0}: Error finding container 10ae15e0c5332779a3d8ca5a2e742705c4434664395e4787fc4b299ad9444477: Status 404 returned error can't find the container with id 10ae15e0c5332779a3d8ca5a2e742705c4434664395e4787fc4b299ad9444477 Oct 31 13:15:23.432673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:23.432579 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:23.950883 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:23.950850 6727 generic.go:356] "Generic (PLEG): container finished" podID="5bb125a0-c715-407f-900f-5ef667b92c05" containerID="46b727204088f8ce199912a364edd3d545c09588479d70838bb82940f4577386" exitCode=0 Oct 31 13:15:23.951062 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:23.950891 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" event={"ID":"5bb125a0-c715-407f-900f-5ef667b92c05","Type":"ContainerDied","Data":"46b727204088f8ce199912a364edd3d545c09588479d70838bb82940f4577386"} Oct 31 13:15:23.951062 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:23.950947 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" event={"ID":"5bb125a0-c715-407f-900f-5ef667b92c05","Type":"ContainerStarted","Data":"10ae15e0c5332779a3d8ca5a2e742705c4434664395e4787fc4b299ad9444477"} Oct 31 13:15:24.190311 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:24.190280 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:24.837010 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:24.836927 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:25.101972 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.101905 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:25.192362 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.192323 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb125a0-c715-407f-900f-5ef667b92c05-kube-api-access-l2n7d\" (UniqueName: \"kubernetes.io/projected/5bb125a0-c715-407f-900f-5ef667b92c05-kube-api-access-l2n7d\") pod \"5bb125a0-c715-407f-900f-5ef667b92c05\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " Oct 31 13:15:25.192572 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.192379 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-openshift-ca\") pod \"5bb125a0-c715-407f-900f-5ef667b92c05\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " Oct 31 13:15:25.192572 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.192503 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-xks-ca\") pod \"5bb125a0-c715-407f-900f-5ef667b92c05\" (UID: \"5bb125a0-c715-407f-900f-5ef667b92c05\") " Oct 31 13:15:25.192826 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.192803 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-openshift-ca" pod "5bb125a0-c715-407f-900f-5ef667b92c05" (UID: "5bb125a0-c715-407f-900f-5ef667b92c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:25.192933 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.192916 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-openshift-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.194628 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.194607 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-xks-ca" pod "5bb125a0-c715-407f-900f-5ef667b92c05" (UID: "5bb125a0-c715-407f-900f-5ef667b92c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:25.194770 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.194753 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb125a0-c715-407f-900f-5ef667b92c05-kube-api-access-l2n7d" pod "5bb125a0-c715-407f-900f-5ef667b92c05" (UID: "5bb125a0-c715-407f-900f-5ef667b92c05"). InnerVolumeSpecName "kube-api-access-l2n7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.293325 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.293294 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2n7d\" (UniqueName: \"kubernetes.io/projected/5bb125a0-c715-407f-900f-5ef667b92c05-kube-api-access-l2n7d\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.293325 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.293316 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb125a0-c715-407f-900f-5ef667b92c05-maas-api-xks-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.541870 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.540199 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:25.962898 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.962861 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" Oct 31 13:15:25.962898 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.962885 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88" event={"ID":"5bb125a0-c715-407f-900f-5ef667b92c05","Type":"ContainerDied","Data":"10ae15e0c5332779a3d8ca5a2e742705c4434664395e4787fc4b299ad9444477"} Oct 31 13:15:25.963368 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:25.962955 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ae15e0c5332779a3d8ca5a2e742705c4434664395e4787fc4b299ad9444477" Oct 31 13:15:26.228227 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:26.228153 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:26.932346 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:26.932273 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:27.655533 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:27.655495 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:28.339599 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:28.339565 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:29.010061 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:29.010028 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:29.729260 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:29.725870 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:30.342158 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:15:30.342127 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:15:30.446245 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:30.446211 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:31.125745 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:31.125710 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:31.818605 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:31.818574 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:32.644451 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:32.644420 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:32.781834 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:32.781806 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:38.209875 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:38.209844 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:38.349923 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:38.349880 6727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88"] Oct 31 13:15:38.356208 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:38.356177 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:38.356911 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:38.356887 6727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-198ea4-blk88"] Oct 31 13:15:38.496021 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:38.495942 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:38.890370 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:38.890337 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:39.298771 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:39.298697 6727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb125a0-c715-407f-900f-5ef667b92c05" path="/var/lib/kubelet/pods/5bb125a0-c715-407f-900f-5ef667b92c05/volumes" Oct 31 13:15:39.654532 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:39.654502 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:15:44.909197 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:44.909157 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:45.080065 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:45.080031 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:45.311588 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:45.311497 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:47.076869 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:47.076836 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:15:52.465661 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:52.465627 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:15:59.628997 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.628958 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck"] Oct 31 13:15:59.629629 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629610 6727 cpu_manager.go:414] "containerMap: removing container" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:59.629736 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629631 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:59.629736 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629679 6727 cpu_manager.go:414] "containerMap: removing container" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:59.629736 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629687 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:59.629736 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629728 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5bb125a0-c715-407f-900f-5ef667b92c05" containerName="revoke-keys" Oct 31 13:15:59.629736 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629736 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb125a0-c715-407f-900f-5ef667b92c05" containerName="revoke-keys" Oct 31 13:15:59.629998 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629852 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d3114d-dff3-46ca-ba25-e051289b7640" containerName="cleanup" Oct 31 13:15:59.629998 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629868 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c85787-503d-4064-9118-6f0393498825" containerName="cleanup" Oct 31 13:15:59.629998 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.629884 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb125a0-c715-407f-900f-5ef667b92c05" containerName="revoke-keys" Oct 31 13:15:59.634174 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.634152 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.641759 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.640787 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:15:59.641759 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.641491 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:15:59.641759 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.641509 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:15:59.641759 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.641575 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x5v7\"" Oct 31 13:15:59.643502 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.643451 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck"] Oct 31 13:15:59.670150 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.670122 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6j8d\" (UniqueName: \"kubernetes.io/projected/a0378e42-81cf-4b8f-9079-151acae5512f-kube-api-access-p6j8d\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.670276 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.670183 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.670276 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.670261 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.771449 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.771420 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6j8d\" (UniqueName: \"kubernetes.io/projected/a0378e42-81cf-4b8f-9079-151acae5512f-kube-api-access-p6j8d\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.771608 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.771516 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.771608 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.771543 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.772143 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.772122 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.773867 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.773848 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.783677 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.783652 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6j8d\" (UniqueName: \"kubernetes.io/projected/a0378e42-81cf-4b8f-9079-151acae5512f-kube-api-access-p6j8d\") pod \"maas-api-revoke-sub-f03629db-r5rck\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:15:59.949031 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:15:59.948969 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:16:00.227320 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:00.227294 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck"] Oct 31 13:16:00.230015 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:16:00.229977 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0378e42_81cf_4b8f_9079_151acae5512f.slice/crio-c8bbb4e2cec2934d17eb321a8916354158dc270e036bae0416b9636c05a1c2ff WatchSource:0}: Error finding container c8bbb4e2cec2934d17eb321a8916354158dc270e036bae0416b9636c05a1c2ff: Status 404 returned error can't find the container with id c8bbb4e2cec2934d17eb321a8916354158dc270e036bae0416b9636c05a1c2ff Oct 31 13:16:00.340865 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:16:00.340831 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:16:01.165595 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:01.165561 6727 generic.go:356] "Generic (PLEG): container finished" podID="a0378e42-81cf-4b8f-9079-151acae5512f" containerID="999ca989b3c605aee29054ca8bc13a6c592fa6a66e358c89c83e4aa884e67b20" exitCode=0 Oct 31 13:16:01.165979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:01.165642 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" event={"ID":"a0378e42-81cf-4b8f-9079-151acae5512f","Type":"ContainerDied","Data":"999ca989b3c605aee29054ca8bc13a6c592fa6a66e358c89c83e4aa884e67b20"} Oct 31 13:16:01.165979 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:01.165702 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" event={"ID":"a0378e42-81cf-4b8f-9079-151acae5512f","Type":"ContainerStarted","Data":"c8bbb4e2cec2934d17eb321a8916354158dc270e036bae0416b9636c05a1c2ff"} Oct 31 13:16:02.301690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.301668 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:16:02.397175 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.397144 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0378e42-81cf-4b8f-9079-151acae5512f-kube-api-access-p6j8d\" (UniqueName: \"kubernetes.io/projected/a0378e42-81cf-4b8f-9079-151acae5512f-kube-api-access-p6j8d\") pod \"a0378e42-81cf-4b8f-9079-151acae5512f\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " Oct 31 13:16:02.397307 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.397194 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-openshift-ca\") pod \"a0378e42-81cf-4b8f-9079-151acae5512f\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " Oct 31 13:16:02.397307 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.397218 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-xks-ca\") pod \"a0378e42-81cf-4b8f-9079-151acae5512f\" (UID: \"a0378e42-81cf-4b8f-9079-151acae5512f\") " Oct 31 13:16:02.397673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.397640 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-openshift-ca" pod "a0378e42-81cf-4b8f-9079-151acae5512f" (UID: "a0378e42-81cf-4b8f-9079-151acae5512f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:02.399258 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.399236 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-xks-ca" pod "a0378e42-81cf-4b8f-9079-151acae5512f" (UID: "a0378e42-81cf-4b8f-9079-151acae5512f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:02.399367 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.399296 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0378e42-81cf-4b8f-9079-151acae5512f-kube-api-access-p6j8d" pod "a0378e42-81cf-4b8f-9079-151acae5512f" (UID: "a0378e42-81cf-4b8f-9079-151acae5512f"). InnerVolumeSpecName "kube-api-access-p6j8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:02.498674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.498612 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6j8d\" (UniqueName: \"kubernetes.io/projected/a0378e42-81cf-4b8f-9079-151acae5512f-kube-api-access-p6j8d\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:02.498674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.498641 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-openshift-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:02.498674 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:02.498652 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0378e42-81cf-4b8f-9079-151acae5512f-maas-api-xks-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:03.177153 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:03.177113 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" event={"ID":"a0378e42-81cf-4b8f-9079-151acae5512f","Type":"ContainerDied","Data":"c8bbb4e2cec2934d17eb321a8916354158dc270e036bae0416b9636c05a1c2ff"} Oct 31 13:16:03.177324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:03.177203 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8bbb4e2cec2934d17eb321a8916354158dc270e036bae0416b9636c05a1c2ff" Oct 31 13:16:03.177324 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:03.177131 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r5rck" Oct 31 13:16:07.391980 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:07.391939 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:09.848991 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:09.848912 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:10.234764 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:10.234651 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:10.464440 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:10.464412 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:16:15.892187 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.892153 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79"] Oct 31 13:16:15.892643 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.892625 6727 cpu_manager.go:414] "containerMap: removing container" podUID="a0378e42-81cf-4b8f-9079-151acae5512f" containerName="revoke-keys" Oct 31 13:16:15.892693 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.892643 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0378e42-81cf-4b8f-9079-151acae5512f" containerName="revoke-keys" Oct 31 13:16:15.892737 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.892708 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0378e42-81cf-4b8f-9079-151acae5512f" containerName="revoke-keys" Oct 31 13:16:15.897222 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.897206 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:15.905487 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.905453 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:16:15.905780 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.905758 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:16:15.905873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.905840 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x5v7\"" Oct 31 13:16:15.905931 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.905910 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:16:15.907043 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.907014 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79"] Oct 31 13:16:15.949411 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.949384 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x748\" (UniqueName: \"kubernetes.io/projected/5435e5db-5b51-48da-aea9-9da3ddc76a9d-kube-api-access-4x748\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:15.949529 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.949424 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:15.949580 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:15.949536 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.050797 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.050771 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x748\" (UniqueName: \"kubernetes.io/projected/5435e5db-5b51-48da-aea9-9da3ddc76a9d-kube-api-access-4x748\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.050894 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.050809 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.050894 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.050835 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.051392 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.051373 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.053132 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.053110 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.063142 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.063122 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x748\" (UniqueName: \"kubernetes.io/projected/5435e5db-5b51-48da-aea9-9da3ddc76a9d-kube-api-access-4x748\") pod \"maas-api-revoke-sub-0165d7d2-hqn79\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.210653 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.210596 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:16.488725 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:16.488630 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79"] Oct 31 13:16:16.491388 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:16:16.491363 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5435e5db_5b51_48da_aea9_9da3ddc76a9d.slice/crio-6fea052271897f5bb3b829eb3de8df55e0e028b1bb5bc694f8cba18937a7ccd3 WatchSource:0}: Error finding container 6fea052271897f5bb3b829eb3de8df55e0e028b1bb5bc694f8cba18937a7ccd3: Status 404 returned error can't find the container with id 6fea052271897f5bb3b829eb3de8df55e0e028b1bb5bc694f8cba18937a7ccd3 Oct 31 13:16:17.253778 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:17.253743 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" event={"ID":"5435e5db-5b51-48da-aea9-9da3ddc76a9d","Type":"ContainerStarted","Data":"47eca9b7bca6fb57dc8a088bc7fc3309fe631e2bd5f0a26d58986922736f0252"} Oct 31 13:16:17.254188 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:17.253794 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" event={"ID":"5435e5db-5b51-48da-aea9-9da3ddc76a9d","Type":"ContainerStarted","Data":"6fea052271897f5bb3b829eb3de8df55e0e028b1bb5bc694f8cba18937a7ccd3"} Oct 31 13:16:17.273009 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:17.272961 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" podStartSLOduration=2.272949067 podStartE2EDuration="2.272949067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:17.271630064 +0000 UTC m=+2888.086644470" watchObservedRunningTime="2026-10-31 13:16:17.272949067 +0000 UTC m=+2888.087963457" Oct 31 13:16:18.260205 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:18.260169 6727 generic.go:356] "Generic (PLEG): container finished" podID="5435e5db-5b51-48da-aea9-9da3ddc76a9d" containerID="47eca9b7bca6fb57dc8a088bc7fc3309fe631e2bd5f0a26d58986922736f0252" exitCode=0 Oct 31 13:16:18.260635 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:18.260257 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" event={"ID":"5435e5db-5b51-48da-aea9-9da3ddc76a9d","Type":"ContainerDied","Data":"47eca9b7bca6fb57dc8a088bc7fc3309fe631e2bd5f0a26d58986922736f0252"} Oct 31 13:16:19.395851 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.395822 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:19.482052 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.482025 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5435e5db-5b51-48da-aea9-9da3ddc76a9d-kube-api-access-4x748\" (UniqueName: \"kubernetes.io/projected/5435e5db-5b51-48da-aea9-9da3ddc76a9d-kube-api-access-4x748\") pod \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " Oct 31 13:16:19.482209 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.482067 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-openshift-ca\") pod \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " Oct 31 13:16:19.482209 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.482114 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-xks-ca\") pod \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\" (UID: \"5435e5db-5b51-48da-aea9-9da3ddc76a9d\") " Oct 31 13:16:19.482534 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.482505 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-openshift-ca" pod "5435e5db-5b51-48da-aea9-9da3ddc76a9d" (UID: "5435e5db-5b51-48da-aea9-9da3ddc76a9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:19.484183 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.484158 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-xks-ca" pod "5435e5db-5b51-48da-aea9-9da3ddc76a9d" (UID: "5435e5db-5b51-48da-aea9-9da3ddc76a9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:19.484301 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.484233 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5435e5db-5b51-48da-aea9-9da3ddc76a9d-kube-api-access-4x748" pod "5435e5db-5b51-48da-aea9-9da3ddc76a9d" (UID: "5435e5db-5b51-48da-aea9-9da3ddc76a9d"). InnerVolumeSpecName "kube-api-access-4x748". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:19.583786 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.583741 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x748\" (UniqueName: \"kubernetes.io/projected/5435e5db-5b51-48da-aea9-9da3ddc76a9d-kube-api-access-4x748\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:19.583786 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.583763 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-openshift-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:19.583786 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:19.583773 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5435e5db-5b51-48da-aea9-9da3ddc76a9d-maas-api-xks-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:20.272262 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:20.272224 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" Oct 31 13:16:20.272262 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:20.272243 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hqn79" event={"ID":"5435e5db-5b51-48da-aea9-9da3ddc76a9d","Type":"ContainerDied","Data":"6fea052271897f5bb3b829eb3de8df55e0e028b1bb5bc694f8cba18937a7ccd3"} Oct 31 13:16:20.272533 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:20.272299 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fea052271897f5bb3b829eb3de8df55e0e028b1bb5bc694f8cba18937a7ccd3" Oct 31 13:16:26.707371 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.707075 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn"] Oct 31 13:16:26.707799 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.707691 6727 cpu_manager.go:414] "containerMap: removing container" podUID="5435e5db-5b51-48da-aea9-9da3ddc76a9d" containerName="revoke-keys" Oct 31 13:16:26.707799 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.707710 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5435e5db-5b51-48da-aea9-9da3ddc76a9d" containerName="revoke-keys" Oct 31 13:16:26.707896 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.707826 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5435e5db-5b51-48da-aea9-9da3ddc76a9d" containerName="revoke-keys" Oct 31 13:16:26.711124 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.711108 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.719212 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.719169 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:16:26.719369 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.719252 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:16:26.719503 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.719372 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x5v7\"" Oct 31 13:16:26.725057 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.721527 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:16:26.725057 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.722916 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn"] Oct 31 13:16:26.842690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.842660 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.842865 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.842737 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.842865 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.842795 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrq5\" (UniqueName: \"kubernetes.io/projected/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-kube-api-access-bzrq5\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.943669 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.943638 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.943669 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.943669 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrq5\" (UniqueName: \"kubernetes.io/projected/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-kube-api-access-bzrq5\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.943892 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.943736 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.944361 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.944335 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.946022 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.946003 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:26.955690 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:26.955672 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrq5\" (UniqueName: \"kubernetes.io/projected/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-kube-api-access-bzrq5\") pod \"maas-api-revoke-sub-6b19338f-7xfjn\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:27.025760 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:27.025709 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:27.331298 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:27.331272 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn"] Oct 31 13:16:27.333762 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:16:27.333731 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6943bc27_2802_4d7b_a73e_3fd8c8af72ab.slice/crio-d62b4ca254d860565e6a12efd8b2198a89825fa67267521d3c3d1500a7a51d65 WatchSource:0}: Error finding container d62b4ca254d860565e6a12efd8b2198a89825fa67267521d3c3d1500a7a51d65: Status 404 returned error can't find the container with id d62b4ca254d860565e6a12efd8b2198a89825fa67267521d3c3d1500a7a51d65 Oct 31 13:16:28.313267 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:28.313235 6727 generic.go:356] "Generic (PLEG): container finished" podID="6943bc27-2802-4d7b-a73e-3fd8c8af72ab" containerID="ba975ecc30516d22eededf4fa6d578a3ed6f572d43666406a57a401ebe038cb5" exitCode=0 Oct 31 13:16:28.313611 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:28.313326 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" event={"ID":"6943bc27-2802-4d7b-a73e-3fd8c8af72ab","Type":"ContainerDied","Data":"ba975ecc30516d22eededf4fa6d578a3ed6f572d43666406a57a401ebe038cb5"} Oct 31 13:16:28.313611 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:28.313376 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" event={"ID":"6943bc27-2802-4d7b-a73e-3fd8c8af72ab","Type":"ContainerStarted","Data":"d62b4ca254d860565e6a12efd8b2198a89825fa67267521d3c3d1500a7a51d65"} Oct 31 13:16:29.452830 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.452801 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:29.568089 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.568061 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-openshift-ca\") pod \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " Oct 31 13:16:29.568266 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.568147 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-xks-ca\") pod \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " Oct 31 13:16:29.568266 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.568170 6727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-kube-api-access-bzrq5\" (UniqueName: \"kubernetes.io/projected/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-kube-api-access-bzrq5\") pod \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\" (UID: \"6943bc27-2802-4d7b-a73e-3fd8c8af72ab\") " Oct 31 13:16:29.568428 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.568384 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-openshift-ca" pod "6943bc27-2802-4d7b-a73e-3fd8c8af72ab" (UID: "6943bc27-2802-4d7b-a73e-3fd8c8af72ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:29.568528 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.568509 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-openshift-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:29.570249 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.570227 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-kube-api-access-bzrq5" pod "6943bc27-2802-4d7b-a73e-3fd8c8af72ab" (UID: "6943bc27-2802-4d7b-a73e-3fd8c8af72ab"). InnerVolumeSpecName "kube-api-access-bzrq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:29.570405 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.570383 6727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-xks-ca" pod "6943bc27-2802-4d7b-a73e-3fd8c8af72ab" (UID: "6943bc27-2802-4d7b-a73e-3fd8c8af72ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:29.669193 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.669166 6727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-maas-api-xks-ca\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:29.669193 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:29.669190 6727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrq5\" (UniqueName: \"kubernetes.io/projected/6943bc27-2802-4d7b-a73e-3fd8c8af72ab-kube-api-access-bzrq5\") on node \"ip-10-0-142-45.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:30.324515 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:30.324487 6727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" Oct 31 13:16:30.324515 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:30.324498 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7xfjn" event={"ID":"6943bc27-2802-4d7b-a73e-3fd8c8af72ab","Type":"ContainerDied","Data":"d62b4ca254d860565e6a12efd8b2198a89825fa67267521d3c3d1500a7a51d65"} Oct 31 13:16:30.324820 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:30.324541 6727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d62b4ca254d860565e6a12efd8b2198a89825fa67267521d3c3d1500a7a51d65" Oct 31 13:16:30.342152 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:16:30.342120 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:16:30.652080 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:30.652001 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:37.520788 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:37.520755 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cbmjj"] Oct 31 13:16:39.153616 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:39.153579 6727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-czj7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:16:39.154053 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:39.153664 6727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-czj7b" podUID="ec9f9199-6032-4a7b-9bcf-37c42a33a41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:16:40.658278 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:40.658236 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:48.636987 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:48.636959 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sbzv6_d96d359f-1732-4408-b38e-e2d0d9143950/manager/0.log" Oct 31 13:16:49.275163 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:49.275139 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zm6w_490fd3c3-446e-40cf-a5ac-ebf98509a290/server/0.log" Oct 31 13:16:50.723717 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:50.723684 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hqn79_5435e5db-5b51-48da-aea9-9da3ddc76a9d/revoke-keys/0.log" Oct 31 13:16:51.227433 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:51.227400 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-7xfjn_6943bc27-2802-4d7b-a73e-3fd8c8af72ab/revoke-keys/0.log" Oct 31 13:16:51.576386 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:51.576361 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-r5rck_a0378e42-81cf-4b8f-9079-151acae5512f/revoke-keys/0.log" Oct 31 13:16:53.783274 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:53.783245 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-r5qv4_495629b0-eb86-4e37-9941-363270905cd1/kuadrant-console-plugin/0.log" Oct 31 13:16:54.201789 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:54.201761 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cbmjj_14f5ae26-97dd-4477-9128-060c0a975982/limitador/0.log" Oct 31 13:16:55.098079 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:55.098041 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-848cd468bfrdfjv_2fc4af62-beef-47b2-85b6-ee18be5cac62/istio-proxy/0.log" Oct 31 13:16:55.277527 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:55.277475 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ff2sl_fee83ef4-beab-4438-80d9-355b9af9e3d2/discovery/0.log" Oct 31 13:16:55.454791 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:55.454724 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-rr2g9_aa2c139d-e9d9-42c2-a78f-c741002c0dd2/kube-auth-proxy/0.log" Oct 31 13:16:56.176088 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:56.176052 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-87q4d_1e6630a4-05af-4601-bc6c-4eced69824f0/istio-proxy/0.log" Oct 31 13:16:56.351034 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:56.350996 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/1.log" Oct 31 13:16:56.693072 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:56.693040 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-hp9kr_e5f06403-3135-46e2-bf0e-7a08611c8d0c/istio-proxy/0.log" Oct 31 13:16:57.579391 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:57.579362 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s_eda0c4ab-b4d2-4b20-9d94-3bb0e414751c/istio-proxy/0.log" Oct 31 13:16:58.520250 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:58.520224 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/0.log" Oct 31 13:16:59.021224 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:59.021195 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr_bfa9cd02-a108-4237-bc59-eb14f393cd2c/main/0.log" Oct 31 13:16:59.382254 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:59.382229 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xczlv_3b468147-6eba-4cbd-aff5-d13da0f35497/main/0.log" Oct 31 13:16:59.731013 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:16:59.730935 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8_528d4d47-3dc0-48e4-bb8c-2fa2b23091f2/main/0.log" Oct 31 13:17:00.076966 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:00.076944 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6_17bb9e9e-099a-4392-87da-ed6a36d161ba/main/0.log" Oct 31 13:17:00.341108 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:17:00.341038 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:17:09.902356 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:09.902321 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:10.259481 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:10.259384 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sbzv6_d96d359f-1732-4408-b38e-e2d0d9143950/manager/0.log" Oct 31 13:17:10.873327 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:10.873291 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zm6w_490fd3c3-446e-40cf-a5ac-ebf98509a290/server/0.log" Oct 31 13:17:11.150216 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:11.150136 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:12.125178 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:12.125153 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hqn79_5435e5db-5b51-48da-aea9-9da3ddc76a9d/revoke-keys/0.log" Oct 31 13:17:12.467611 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:12.467521 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-7xfjn_6943bc27-2802-4d7b-a73e-3fd8c8af72ab/revoke-keys/0.log" Oct 31 13:17:12.823363 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:12.823337 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-r5rck_a0378e42-81cf-4b8f-9079-151acae5512f/revoke-keys/0.log" Oct 31 13:17:14.646722 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:14.646691 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-r5qv4_495629b0-eb86-4e37-9941-363270905cd1/kuadrant-console-plugin/0.log" Oct 31 13:17:15.017048 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:15.016968 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cbmjj_14f5ae26-97dd-4477-9128-060c0a975982/limitador/0.log" Oct 31 13:17:15.749055 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:15.749025 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-848cd468bfrdfjv_2fc4af62-beef-47b2-85b6-ee18be5cac62/istio-proxy/0.log" Oct 31 13:17:15.913436 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:15.913410 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ff2sl_fee83ef4-beab-4438-80d9-355b9af9e3d2/discovery/0.log" Oct 31 13:17:16.087859 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:16.087834 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-rr2g9_aa2c139d-e9d9-42c2-a78f-c741002c0dd2/kube-auth-proxy/0.log" Oct 31 13:17:16.784667 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:16.784635 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-87q4d_1e6630a4-05af-4601-bc6c-4eced69824f0/istio-proxy/0.log" Oct 31 13:17:16.956254 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:16.956223 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/1.log" Oct 31 13:17:17.306074 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:17.306041 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-hp9kr_e5f06403-3135-46e2-bf0e-7a08611c8d0c/istio-proxy/0.log" Oct 31 13:17:18.158820 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:18.158789 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s_eda0c4ab-b4d2-4b20-9d94-3bb0e414751c/istio-proxy/0.log" Oct 31 13:17:19.212937 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:19.212909 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5rbhr_bfa9cd02-a108-4237-bc59-eb14f393cd2c/main/0.log" Oct 31 13:17:19.565701 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:19.565628 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xczlv_3b468147-6eba-4cbd-aff5-d13da0f35497/main/0.log" Oct 31 13:17:19.902322 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:19.902295 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m5hw8_528d4d47-3dc0-48e4-bb8c-2fa2b23091f2/main/0.log" Oct 31 13:17:20.239445 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:20.239366 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-jprb6_17bb9e9e-099a-4392-87da-ed6a36d161ba/main/0.log" Oct 31 13:17:20.485720 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:20.485691 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:20.624547 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:17:20.624513 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:30.341256 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:17:30.341222 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:18:00.340718 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:18:00.340679 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:18:09.956848 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:09.956819 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:10.336227 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:10.336203 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:30.340854 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:18:30.340818 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:18:34.754995 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:34.754956 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:40.698904 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:40.698868 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:40.853791 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:40.853757 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:44.025670 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:44.025642 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g7zbh_a166dd5f-2e0c-45a0-8dc6-692741a8e1b3/global-pull-secret-syncer/0.log" Oct 31 13:18:44.252536 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:44.252509 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" Oct 31 13:18:44.437586 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:44.437557 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-45.us-east-2.compute.internal_2e9561195d04d47702d33d861addb8e7/haproxy/0.log" Oct 31 13:18:51.659247 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:51.659222 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-r5qv4_495629b0-eb86-4e37-9941-363270905cd1/kuadrant-console-plugin/0.log" Oct 31 13:18:51.749353 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:51.749326 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cbmjj_14f5ae26-97dd-4477-9128-060c0a975982/limitador/0.log" Oct 31 13:18:53.110627 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.110596 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f94000a6-09f2-4acb-8538-99d495b050f1/alertmanager/0.log" Oct 31 13:18:53.146204 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.146175 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f94000a6-09f2-4acb-8538-99d495b050f1/config-reloader/0.log" Oct 31 13:18:53.173316 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.173295 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f94000a6-09f2-4acb-8538-99d495b050f1/kube-rbac-proxy-web/0.log" Oct 31 13:18:53.199073 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.199056 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f94000a6-09f2-4acb-8538-99d495b050f1/kube-rbac-proxy/0.log" Oct 31 13:18:53.226503 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.226487 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f94000a6-09f2-4acb-8538-99d495b050f1/kube-rbac-proxy-metric/0.log" Oct 31 13:18:53.262643 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.262623 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f94000a6-09f2-4acb-8538-99d495b050f1/prom-label-proxy/0.log" Oct 31 13:18:53.298196 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.298159 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f94000a6-09f2-4acb-8538-99d495b050f1/init-config-reloader/0.log" Oct 31 13:18:53.394555 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.394483 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-gbwhw_7c0a973f-9d43-4bec-a134-eda7d2b7a052/kube-state-metrics/0.log" Oct 31 13:18:53.418655 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.418634 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-gbwhw_7c0a973f-9d43-4bec-a134-eda7d2b7a052/kube-rbac-proxy-main/0.log" Oct 31 13:18:53.444742 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.444709 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-gbwhw_7c0a973f-9d43-4bec-a134-eda7d2b7a052/kube-rbac-proxy-self/0.log" Oct 31 13:18:53.516838 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.516816 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-wqp22_0d06e6e9-51c0-403a-8ea0-ecc9de005803/monitoring-plugin/0.log" Oct 31 13:18:53.647321 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.647264 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67z7l_d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/node-exporter/0.log" Oct 31 13:18:53.675615 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.675593 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67z7l_d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/kube-rbac-proxy/0.log" Oct 31 13:18:53.705968 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.705946 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67z7l_d5fe0d0c-038c-4ac2-bbf3-fc5881fe3105/init-textfile/0.log" Oct 31 13:18:53.836173 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.836146 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bp6vl_a95a2e68-c596-4c2e-b1bd-20d31bb31919/kube-rbac-proxy-main/0.log" Oct 31 13:18:53.862191 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.862169 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bp6vl_a95a2e68-c596-4c2e-b1bd-20d31bb31919/kube-rbac-proxy-self/0.log" Oct 31 13:18:53.896437 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.896415 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bp6vl_a95a2e68-c596-4c2e-b1bd-20d31bb31919/openshift-state-metrics/0.log" Oct 31 13:18:53.940648 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.940578 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49e534f-01f6-4644-9151-c7867847496d/prometheus/0.log" Oct 31 13:18:53.969576 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:53.969558 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49e534f-01f6-4644-9151-c7867847496d/config-reloader/0.log" Oct 31 13:18:54.001020 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.001003 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49e534f-01f6-4644-9151-c7867847496d/thanos-sidecar/0.log" Oct 31 13:18:54.027830 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.027804 6727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25"] Oct 31 13:18:54.028232 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.028218 6727 cpu_manager.go:414] "containerMap: removing container" podUID="6943bc27-2802-4d7b-a73e-3fd8c8af72ab" containerName="revoke-keys" Oct 31 13:18:54.028284 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.028233 6727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6943bc27-2802-4d7b-a73e-3fd8c8af72ab" containerName="revoke-keys" Oct 31 13:18:54.028330 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.028300 6727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6943bc27-2802-4d7b-a73e-3fd8c8af72ab" containerName="revoke-keys" Oct 31 13:18:54.031937 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.031915 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.036823 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.036799 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q58mx\"/\"kube-root-ca.crt\"" Oct 31 13:18:54.037673 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.037652 6727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q58mx\"/\"default-dockercfg-7dg4w\"" Oct 31 13:18:54.037791 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.037689 6727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q58mx\"/\"openshift-service-ca.crt\"" Oct 31 13:18:54.045645 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.045620 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25"] Oct 31 13:18:54.054256 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.054235 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49e534f-01f6-4644-9151-c7867847496d/kube-rbac-proxy-web/0.log" Oct 31 13:18:54.090678 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.090656 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49e534f-01f6-4644-9151-c7867847496d/kube-rbac-proxy/0.log" Oct 31 13:18:54.127450 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.127422 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49e534f-01f6-4644-9151-c7867847496d/kube-rbac-proxy-thanos/0.log" Oct 31 13:18:54.151873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.151843 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-sys\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.151873 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.151876 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-proc\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.152128 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.151934 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fxs\" (UniqueName: \"kubernetes.io/projected/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-kube-api-access-g9fxs\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.152128 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.152007 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-podres\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.152128 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.152100 6727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-lib-modules\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.160186 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.160163 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49e534f-01f6-4644-9151-c7867847496d/init-config-reloader/0.log" Oct 31 13:18:54.253746 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253666 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-lib-modules\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.253908 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253770 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-sys\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.253908 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253792 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-proc\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.253908 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253806 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-lib-modules\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.253908 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253814 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fxs\" (UniqueName: \"kubernetes.io/projected/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-kube-api-access-g9fxs\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.253908 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253885 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-proc\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.253908 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253885 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-sys\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.254143 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253903 6727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-podres\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.254143 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.253952 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-podres\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.265672 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.265644 6727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fxs\" (UniqueName: \"kubernetes.io/projected/987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b-kube-api-access-g9fxs\") pod \"perf-node-gather-daemonset-kxm25\" (UID: \"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.344920 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.344867 6727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:54.384289 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.384249 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-577cf77cd5-p495x_d2e7f16d-ec14-40bd-908f-272016e1464f/thanos-query/0.log" Oct 31 13:18:54.415797 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.415763 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-577cf77cd5-p495x_d2e7f16d-ec14-40bd-908f-272016e1464f/kube-rbac-proxy-web/0.log" Oct 31 13:18:54.442396 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.442367 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-577cf77cd5-p495x_d2e7f16d-ec14-40bd-908f-272016e1464f/kube-rbac-proxy/0.log" Oct 31 13:18:54.473309 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.473282 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-577cf77cd5-p495x_d2e7f16d-ec14-40bd-908f-272016e1464f/prom-label-proxy/0.log" Oct 31 13:18:54.499383 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.499358 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-577cf77cd5-p495x_d2e7f16d-ec14-40bd-908f-272016e1464f/kube-rbac-proxy-rules/0.log" Oct 31 13:18:54.525337 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.525282 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-577cf77cd5-p495x_d2e7f16d-ec14-40bd-908f-272016e1464f/kube-rbac-proxy-metrics/0.log" Oct 31 13:18:54.618719 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:54.618684 6727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25"] Oct 31 13:18:54.622078 ip-10-0-142-45 kubenswrapper[6727]: W1031 13:18:54.622052 6727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod987f6b07_9b4e_4fc1_a9a4_d6f1bed3ca3b.slice/crio-0e32ad503fe0c9e013c5457074bbba3f7ab73a9d18bbb906d886368f814689a4 WatchSource:0}: Error finding container 0e32ad503fe0c9e013c5457074bbba3f7ab73a9d18bbb906d886368f814689a4: Status 404 returned error can't find the container with id 0e32ad503fe0c9e013c5457074bbba3f7ab73a9d18bbb906d886368f814689a4 Oct 31 13:18:55.036419 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:55.036382 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" event={"ID":"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b","Type":"ContainerStarted","Data":"efca2ea27c3079a2190bbeb2c2b17a0dddc1e14e2434e4ae115ac82cf991296f"} Oct 31 13:18:55.036635 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:55.036444 6727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" event={"ID":"987f6b07-9b4e-4fc1-a9a4-d6f1bed3ca3b","Type":"ContainerStarted","Data":"0e32ad503fe0c9e013c5457074bbba3f7ab73a9d18bbb906d886368f814689a4"} Oct 31 13:18:55.036635 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:55.036595 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:18:55.058385 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:55.058340 6727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" podStartSLOduration=1.058329297 podStartE2EDuration="1.058329297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:55.056198132 +0000 UTC m=+3045.871212535" watchObservedRunningTime="2026-10-31 13:18:55.058329297 +0000 UTC m=+3045.873343687" Oct 31 13:18:58.533752 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:58.533719 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4jx6k_052e1a61-3220-4a91-83c5-a08bb829f358/dns/0.log" Oct 31 13:18:58.561163 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:58.561127 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4jx6k_052e1a61-3220-4a91-83c5-a08bb829f358/kube-rbac-proxy/0.log" Oct 31 13:18:58.770300 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:58.770274 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjlsm_6d278316-af3e-4b36-8505-a5d464016e85/dns-node-resolver/0.log" Oct 31 13:18:59.432737 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:18:59.432714 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dz7qv_8fc0c5ec-9758-40ff-9990-ab568b8d02dd/node-ca/0.log" Oct 31 13:19:00.342645 ip-10-0-142-45 kubenswrapper[6727]: E1031 13:19:00.342610 6727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6x24" containerName="extractor" Oct 31 13:19:00.589028 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:00.588998 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-848cd468bfrdfjv_2fc4af62-beef-47b2-85b6-ee18be5cac62/istio-proxy/0.log" Oct 31 13:19:01.052032 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.052002 6727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-kxm25" Oct 31 13:19:01.291602 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.291579 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ff2sl_fee83ef4-beab-4438-80d9-355b9af9e3d2/discovery/0.log" Oct 31 13:19:01.325263 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.325241 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-rr2g9_aa2c139d-e9d9-42c2-a78f-c741002c0dd2/kube-auth-proxy/0.log" Oct 31 13:19:01.472531 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.472512 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-87q4d_1e6630a4-05af-4601-bc6c-4eced69824f0/istio-proxy/0.log" Oct 31 13:19:01.504841 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.504820 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/0.log" Oct 31 13:19:01.505744 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.505725 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/1.log" Oct 31 13:19:01.575556 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.575509 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-hp9kr_e5f06403-3135-46e2-bf0e-7a08611c8d0c/istio-proxy/0.log" Oct 31 13:19:01.747554 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:01.747536 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s_eda0c4ab-b4d2-4b20-9d94-3bb0e414751c/istio-proxy/0.log" Oct 31 13:19:04.166720 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:04.166691 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jlbzw_c41fab2e-6e79-4fe1-a46e-996b4012efdf/serve-healthcheck-canary/0.log" Oct 31 13:19:04.370741 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:04.370706 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-848cd468bfrdfjv_2fc4af62-beef-47b2-85b6-ee18be5cac62/istio-proxy/0.log" Oct 31 13:19:04.903964 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:04.903938 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b6x24_2726bde3-96ae-4717-8732-f7e792a0ddfb/kube-rbac-proxy/0.log" Oct 31 13:19:04.932444 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:04.932413 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b6x24_2726bde3-96ae-4717-8732-f7e792a0ddfb/exporter/0.log" Oct 31 13:19:04.959370 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:04.959346 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b6x24_2726bde3-96ae-4717-8732-f7e792a0ddfb/extractor/0.log" Oct 31 13:19:05.052718 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:05.052695 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ff2sl_fee83ef4-beab-4438-80d9-355b9af9e3d2/discovery/0.log" Oct 31 13:19:05.063251 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:05.063230 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-rr2g9_aa2c139d-e9d9-42c2-a78f-c741002c0dd2/kube-auth-proxy/0.log" Oct 31 13:19:05.165870 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:05.165814 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-87q4d_1e6630a4-05af-4601-bc6c-4eced69824f0/istio-proxy/0.log" Oct 31 13:19:05.188253 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:05.188227 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/0.log" Oct 31 13:19:05.197221 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:05.197203 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/1.log" Oct 31 13:19:05.240060 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:05.240041 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-hp9kr_e5f06403-3135-46e2-bf0e-7a08611c8d0c/istio-proxy/0.log" Oct 31 13:19:05.351705 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:05.351683 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s_eda0c4ab-b4d2-4b20-9d94-3bb0e414751c/istio-proxy/0.log" Oct 31 13:19:07.763937 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:07.763900 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5967ccc7d4-nj6n9_aa911761-d836-4067-a704-515f3158e830/manager/0.log" Oct 31 13:19:07.797697 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:07.797674 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5967ccc7d4-nj6n9_aa911761-d836-4067-a704-515f3158e830/copy-manifests/0.log" Oct 31 13:19:08.314738 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:08.314707 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-848cd468bfrdfjv_2fc4af62-beef-47b2-85b6-ee18be5cac62/istio-proxy/0.log" Oct 31 13:19:08.981210 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:08.981183 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ff2sl_fee83ef4-beab-4438-80d9-355b9af9e3d2/discovery/0.log" Oct 31 13:19:09.010617 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:09.010595 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-rr2g9_aa2c139d-e9d9-42c2-a78f-c741002c0dd2/kube-auth-proxy/0.log" Oct 31 13:19:09.090204 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:09.090175 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-87q4d_1e6630a4-05af-4601-bc6c-4eced69824f0/istio-proxy/0.log" Oct 31 13:19:09.118272 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:09.118245 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/0.log" Oct 31 13:19:09.119238 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:09.119214 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-czj7b_ec9f9199-6032-4a7b-9bcf-37c42a33a41a/istio-proxy/1.log" Oct 31 13:19:09.165441 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:09.165419 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-hp9kr_e5f06403-3135-46e2-bf0e-7a08611c8d0c/istio-proxy/0.log" Oct 31 13:19:09.267685 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:09.267614 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-fdb9847-dbj7s_eda0c4ab-b4d2-4b20-9d94-3bb0e414751c/istio-proxy/0.log" Oct 31 13:19:09.988564 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:09.988529 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sbzv6_d96d359f-1732-4408-b38e-e2d0d9143950/manager/0.log" Oct 31 13:19:10.011040 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:10.011012 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:10.400791 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:10.400763 6727 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6b687_32afbb71-9024-4d12-a27a-994892cf38e4/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:10.703279 ip-10-0-142-45 kubenswrapper[6727]: I1031 13:19:10.703211 6727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6zm6w_490fd3c3-446e-40cf-a5ac-ebf98509a290/server/0.log"