Sep 10 19:10:32.616713 ip-10-0-143-43 systemd[1]: Starting Kubernetes Kubelet... Sep 10 19:10:33.054384 ip-10-0-143-43 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:10:33.054384 ip-10-0-143-43 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 19:10:33.054384 ip-10-0-143-43 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:10:33.054384 ip-10-0-143-43 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 19:10:33.054384 ip-10-0-143-43 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:10:33.055709 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.055549 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 19:10:33.058998 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.058983 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:10:33.058998 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.058998 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059002 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059006 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059009 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059012 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059014 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059019 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059022 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059025 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059028 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059030 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059032 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059035 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059038 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059041 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059043 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059046 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059049 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059051 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:10:33.059061 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059054 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059056 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059062 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059065 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059068 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059071 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059073 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059078 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059082 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059085 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059088 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059090 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059093 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059095 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059098 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059101 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059103 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059105 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059108 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059110 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:10:33.059563 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059113 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059115 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059117 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059120 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059123 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059125 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059128 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059130 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059132 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059135 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059137 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059139 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059142 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059145 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059149 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059152 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059155 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059158 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059160 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:10:33.060045 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059162 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059165 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059168 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059171 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059173 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059176 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059178 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059181 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059183 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059185 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059188 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059190 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059192 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059195 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059197 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059199 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059202 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059204 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059207 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059210 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:10:33.060518 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059212 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059215 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059219 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059222 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059225 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059228 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059230 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059233 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059237 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059240 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059242 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059245 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059247 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059250 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059252 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059254 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059257 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059259 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059262 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:10:33.060995 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.059264 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:10:33.061953 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061941 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:10:33.061953 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061952 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061955 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061958 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061961 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061964 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061966 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061970 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061972 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061975 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061977 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061981 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061985 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061988 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061992 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061996 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.061999 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062002 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062005 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062008 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:10:33.062010 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062011 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062014 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062017 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062020 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062023 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062026 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062028 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062031 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062034 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062036 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062039 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062042 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062044 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062046 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062049 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062051 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062054 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062057 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062059 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:10:33.062543 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062062 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062064 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062067 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062069 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062071 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062074 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062076 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062078 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062081 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062084 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062086 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062089 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062091 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062094 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062097 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062100 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062102 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062105 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062108 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062110 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:10:33.063028 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062113 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062115 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062118 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062120 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062123 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062125 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062128 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062131 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062134 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062137 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062140 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062143 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062146 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062149 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062151 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062154 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062156 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062159 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062161 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062164 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:10:33.063530 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062167 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062169 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062172 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062174 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062177 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062179 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062182 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062186 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062188 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062191 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062193 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062196 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062199 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062201 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062203 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062206 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062208 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062210 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062213 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:10:33.064007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062215 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062648 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062654 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062658 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062661 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062664 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062667 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062670 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062672 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062675 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062678 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062680 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062683 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062685 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062689 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062691 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062694 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062696 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062700 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062705 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:10:33.064487 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062708 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062711 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062714 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062716 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062719 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062721 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062724 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062726 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062729 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062731 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062733 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062736 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062738 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062740 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062743 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062746 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062748 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062750 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062753 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062755 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:10:33.064971 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062758 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062760 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062763 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062765 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062768 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062771 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062775 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062778 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062781 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062783 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062785 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062788 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062791 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062794 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062797 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062799 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062802 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062804 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062807 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:10:33.065479 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062809 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062812 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062814 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062816 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062820 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062823 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062826 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062829 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062832 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062834 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062837 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062839 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062842 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062845 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062847 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062850 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062852 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062855 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062857 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062860 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:10:33.066031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062863 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062866 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062868 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062872 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062875 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062878 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062880 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062882 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062885 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062888 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062890 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062893 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062895 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062898 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062900 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062903 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062906 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062908 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062911 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062913 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.062916 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:10:33.066542 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.062990 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063023 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063029 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063034 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063038 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063042 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063046 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063051 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063054 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063057 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063060 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063063 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063066 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063070 2576 flags.go:64] FLAG: --cgroup-root="" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063073 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063076 2576 flags.go:64] FLAG: --client-ca-file="" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063079 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063082 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063086 2576 flags.go:64] FLAG: --cluster-domain="" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063089 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063092 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063095 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063098 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063102 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 19:10:33.067052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063104 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063107 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063110 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063113 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063116 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063119 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063122 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063125 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063128 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063131 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063134 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063137 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063142 2576 flags.go:64] FLAG: --enable-server="true" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063144 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063149 2576 flags.go:64] FLAG: --event-burst="100" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063152 2576 flags.go:64] FLAG: --event-qps="50" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063155 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063158 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063160 2576 flags.go:64] FLAG: --eviction-hard="" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063164 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063167 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063170 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063173 2576 flags.go:64] FLAG: --eviction-soft="" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063176 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063179 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 19:10:33.067633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063182 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063185 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063187 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063190 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063193 2576 flags.go:64] FLAG: --feature-gates="" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063196 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063199 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063202 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063205 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063208 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063211 2576 flags.go:64] FLAG: --help="false" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063214 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063217 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063220 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063223 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063226 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063229 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063232 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063234 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063238 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063241 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063244 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063247 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063250 2576 flags.go:64] FLAG: --kube-reserved="" Sep 10 19:10:33.068346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063252 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063255 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063258 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063261 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063263 2576 flags.go:64] FLAG: --lock-file="" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063266 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063269 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063272 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063277 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063280 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063283 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063285 2576 flags.go:64] FLAG: --logging-format="text" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063288 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063291 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063294 2576 flags.go:64] FLAG: --manifest-url="" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063297 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063301 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063304 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063307 2576 flags.go:64] FLAG: --max-pods="110" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063310 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063313 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063316 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063319 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063321 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063324 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 19:10:33.068971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063327 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063333 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063337 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063340 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063343 2576 flags.go:64] FLAG: --pod-cidr="" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063347 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063353 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063356 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063359 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063362 2576 flags.go:64] FLAG: --port="10250" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063365 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063367 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-079f08ef90fdebc75" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063370 2576 flags.go:64] FLAG: --qos-reserved="" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063373 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063377 2576 flags.go:64] FLAG: --register-node="true" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063380 2576 flags.go:64] FLAG: --register-with-taints="" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063383 2576 flags.go:64] FLAG: --registry-burst="10" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063386 2576 flags.go:64] FLAG: --registry-qps="5" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063388 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063391 2576 flags.go:64] FLAG: --reserved-memory="" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063394 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063397 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063400 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063402 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063405 2576 flags.go:64] FLAG: --runonce="false" Sep 10 19:10:33.069593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063408 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063411 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063430 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063434 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063437 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063440 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063443 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063445 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063448 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063451 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063455 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063458 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063463 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063466 2576 flags.go:64] FLAG: --system-cgroups="" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063468 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063474 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063476 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063479 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063483 2576 flags.go:64] FLAG: --tls-min-version="" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063485 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063488 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063491 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063500 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063503 2576 flags.go:64] FLAG: --v="2" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063507 2576 flags.go:64] FLAG: --version="false" Sep 10 19:10:33.070234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063511 2576 flags.go:64] FLAG: --vmodule="" Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063514 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063517 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063623 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063627 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063630 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063632 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063635 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063638 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063640 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063642 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063645 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063648 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063650 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063653 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063655 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063658 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063662 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063664 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063668 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:10:33.070904 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063670 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063673 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063675 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063678 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063680 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063684 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063687 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063690 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063692 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063695 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063698 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063700 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063703 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063705 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063708 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063710 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063712 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063715 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063717 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:10:33.071513 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063720 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063722 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063724 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063727 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063729 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063732 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063734 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063737 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063739 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063742 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063745 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063748 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063751 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063754 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063756 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063759 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063761 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063764 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063767 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063769 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:10:33.071985 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063773 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063776 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063779 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063783 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063786 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063789 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063792 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063795 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063797 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063800 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063802 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063805 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063807 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063810 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063812 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063814 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063817 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063821 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063823 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:10:33.072502 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063826 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063829 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063832 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063835 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063838 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063841 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063844 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063847 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063849 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063852 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063854 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063857 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063859 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063862 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063864 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063866 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063869 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063871 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063874 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:10:33.072968 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063876 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063879 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063881 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063884 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.063886 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.063891 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.071769 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.071786 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.072700 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.072708 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072766 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072772 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072775 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072777 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072780 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:10:33.073443 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072783 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072786 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072789 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072792 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072795 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072798 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072800 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072803 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072805 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072808 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072810 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072813 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072816 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072818 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072820 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072823 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072826 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072828 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072831 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:10:33.073821 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072834 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072838 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072841 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072844 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072848 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072853 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072856 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072860 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072862 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072865 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072867 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072870 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072873 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072875 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072878 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072880 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072883 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072885 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072888 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:10:33.074277 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072890 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072893 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072896 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072898 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072901 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072904 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072906 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072909 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072911 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072914 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072916 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072919 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072921 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072923 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072926 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072928 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072931 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072933 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072936 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072939 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:10:33.074827 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072942 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072945 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072947 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072950 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072952 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072956 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072959 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072962 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072965 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072967 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072970 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072973 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072975 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072978 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072981 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072984 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072986 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072989 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072991 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:10:33.075340 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072994 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072996 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.072998 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073002 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073004 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073007 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073009 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073012 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073014 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073017 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073019 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073022 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073025 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073028 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073031 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073033 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:10:33.075813 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073036 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.073041 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073159 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073164 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073168 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073171 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073174 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073176 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073179 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073182 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073185 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073188 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073190 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073193 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073197 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:10:33.076192 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073199 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073202 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073204 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073207 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073209 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073211 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073215 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073218 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073220 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073223 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073225 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073228 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073230 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073233 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073236 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073239 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073242 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073244 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073246 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073249 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:10:33.076574 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073251 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073253 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073256 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073260 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073263 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073266 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073268 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073272 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073275 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073278 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073280 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073283 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073285 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073288 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073290 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073292 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073295 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073297 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073299 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:10:33.077056 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073302 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073305 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073307 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073309 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073312 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073314 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073317 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073319 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073322 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073325 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073327 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073330 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073332 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073334 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073337 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073339 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073342 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073344 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073347 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073349 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:10:33.077536 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073352 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073354 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073357 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073361 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073364 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073367 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073370 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073372 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073375 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073378 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073380 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073382 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073385 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073387 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073390 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073392 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073395 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073397 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073400 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073402 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:10:33.078011 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073405 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073408 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073411 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073413 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073428 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073431 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:33.073433 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.073438 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.073552 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.076072 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.077195 2576 server.go:1019] "Starting client certificate rotation" Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.077302 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 19:10:33.078509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.077339 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 19:10:33.080216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.080201 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 19:10:33.081574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.081553 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 19:10:33.090055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.090038 2576 log.go:25] "Validated CRI v1 runtime API" Sep 10 19:10:33.093571 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.093554 2576 log.go:25] "Validated CRI v1 image API" Sep 10 19:10:33.094460 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.094443 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 19:10:33.098076 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.098053 2576 fs.go:135] Filesystem UUIDs: map[716d7530-0526-4c3c-8e1d-32463dc5d655:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a707993d-148c-4063-b4ea-837ce3de5d0b:/dev/nvme0n1p3] Sep 10 19:10:33.098136 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.098074 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 19:10:33.104642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.104522 2576 manager.go:216] Machine: {Timestamp:2026-09-10 19:10:33.103023612 +0000 UTC m=+0.369633826 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198300 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dd2aa63fd7b53438b51bf04001e58 SystemUUID:ec2dd2aa-63fd-7b53-438b-51bf04001e58 BootID:90015781-16ac-47e2-97b1-f61a7d7b0328 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:57:12:03:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:57:12:03:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:83:d7:2e:26:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 19:10:33.104642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.104627 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 19:10:33.104773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.104739 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 19:10:33.108225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.108195 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 19:10:33.108363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.108229 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 19:10:33.109089 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.109077 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 19:10:33.109122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.109092 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 19:10:33.109122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.109100 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 19:10:33.109122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.109114 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 19:10:33.110453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.110443 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 19:10:33.110832 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.110814 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 19:10:33.115201 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115189 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 19:10:33.115327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115318 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 19:10:33.115356 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115349 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 10 19:10:33.115391 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115358 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 19:10:33.115391 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115374 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 19:10:33.115391 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115384 2576 kubelet.go:398] "Adding apiserver pod source" Sep 10 19:10:33.115522 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115394 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 19:10:33.115869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115850 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 19:10:33.115959 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.115876 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 19:10:33.118666 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.118648 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 19:10:33.121951 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.121935 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 19:10:33.122038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.121967 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 19:10:33.123380 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123369 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123385 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123391 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123397 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123405 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123410 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123438 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123444 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 19:10:33.123449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123451 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 19:10:33.123689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123457 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 19:10:33.123689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123465 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 19:10:33.123689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.123470 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 19:10:33.124358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.124344 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 19:10:33.124358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.124355 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 19:10:33.126960 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.126915 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 19:10:33.127695 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.127672 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 19:10:33.127695 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.127672 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 19:10:33.128314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.128298 2576 server.go:1268] "Started kubelet" Sep 10 19:10:33.128383 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.128363 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 19:10:33.128624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.128515 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 19:10:33.128624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.128582 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 19:10:33.129181 ip-10-0-143-43 systemd[1]: Started Kubernetes Kubelet. Sep 10 19:10:33.133974 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.133838 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 19:10:33.135812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.135631 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 10 19:10:33.136257 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.136236 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 19:10:33.139379 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.139362 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 19:10:33.139795 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.138631 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-43.ec2.internal.18d40b59fed48e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-43.ec2.internal,UID:ip-10-0-143-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-43.ec2.internal,},FirstTimestamp:2026-09-10 19:10:33.128267392 +0000 UTC m=+0.394877606,LastTimestamp:2026-09-10 19:10:33.128267392 +0000 UTC m=+0.394877606,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-43.ec2.internal,}" Sep 10 19:10:33.140062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140044 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 19:10:33.140378 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.140356 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.140866 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140847 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 19:10:33.140866 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140869 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 19:10:33.140995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140887 2576 factory.go:55] Registering systemd factory Sep 10 19:10:33.140995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140902 2576 factory.go:223] Registration of the systemd container factory successfully Sep 10 19:10:33.140995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140849 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 19:10:33.140995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140948 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 10 19:10:33.140995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.140958 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 10 19:10:33.141164 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.141108 2576 factory.go:153] Registering CRI-O factory Sep 10 19:10:33.141164 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.141120 2576 factory.go:223] Registration of the crio container factory successfully Sep 10 19:10:33.141236 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.141176 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 19:10:33.141236 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.141195 2576 factory.go:103] Registering Raw factory Sep 10 19:10:33.141236 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.141210 2576 manager.go:1196] Started watching for new ooms in manager Sep 10 19:10:33.142368 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.142351 2576 manager.go:349] Starting recovery of all containers Sep 10 19:10:33.146484 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.146453 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 19:10:33.146484 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.146458 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 19:10:33.152215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.152191 2576 manager.go:354] Recovery completed Sep 10 19:10:33.157118 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.157104 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:33.159104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.159081 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:33.159167 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.159120 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:33.159167 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.159131 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:33.159673 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.159659 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 19:10:33.159673 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.159672 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 19:10:33.159756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.159676 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:33.159756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.159687 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 19:10:33.160358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.160342 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:33.160453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.160368 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:33.160453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.160379 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:33.161348 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.161278 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-43.ec2.internal.18d40b5a00ab13e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-43.ec2.internal,UID:ip-10-0-143-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-43.ec2.internal,},FirstTimestamp:2026-09-10 19:10:33.159103462 +0000 UTC m=+0.425713674,LastTimestamp:2026-09-10 19:10:33.159103462 +0000 UTC m=+0.425713674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-43.ec2.internal,}" Sep 10 19:10:33.161815 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.161802 2576 policy_none.go:49] "None policy: Start" Sep 10 19:10:33.161858 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.161818 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 19:10:33.161858 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.161828 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 19:10:33.163760 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.163748 2576 policy_none.go:47] "Start" Sep 10 19:10:33.170379 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.170352 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qpv6f" Sep 10 19:10:33.172193 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.172126 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-43.ec2.internal.18d40b5a00ab6a23 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-43.ec2.internal,UID:ip-10-0-143-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-43.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-43.ec2.internal,},FirstTimestamp:2026-09-10 19:10:33.159125539 +0000 UTC m=+0.425735751,LastTimestamp:2026-09-10 19:10:33.159125539 +0000 UTC m=+0.425735751,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-43.ec2.internal,}" Sep 10 19:10:33.178475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.178455 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qpv6f" Sep 10 19:10:33.181995 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.181930 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-43.ec2.internal.18d40b5a00ab8f6a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-43.ec2.internal,UID:ip-10-0-143-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-43.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-43.ec2.internal,},FirstTimestamp:2026-09-10 19:10:33.159135082 +0000 UTC m=+0.425745296,LastTimestamp:2026-09-10 19:10:33.159135082 +0000 UTC m=+0.425745296,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-43.ec2.internal,}" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.210976 2576 manager.go:337] "Starting Device Plugin manager" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.211018 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.211032 2576 server.go:87] "Starting device plugin registration server" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.211321 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.211332 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.211447 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.211531 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.211540 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.212103 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 19:10:33.218724 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.212156 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.244151 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.244123 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 19:10:33.245390 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.245376 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 19:10:33.245477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.245446 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 19:10:33.245532 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.245478 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 19:10:33.245532 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.245524 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 19:10:33.247881 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.247855 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:10:33.311715 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.311627 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:33.313658 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.313632 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:33.313778 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.313666 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:33.313778 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.313680 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:33.313778 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.313706 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.323077 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.323049 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.323185 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.323080 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-43.ec2.internal\": node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.346357 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.346326 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal"] Sep 10 19:10:33.346483 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.346446 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:33.347451 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.347432 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:33.347565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.347467 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:33.347565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.347480 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:33.349952 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.349938 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:33.350087 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.350131 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350117 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:33.350574 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.350557 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.350773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350759 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:33.350814 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350788 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:33.350814 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350799 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:33.350887 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350758 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:33.350887 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350860 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:33.350887 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.350871 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:33.353171 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.353156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.353255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.353181 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:33.353928 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.353911 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:33.354018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.353940 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:33.354018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.353954 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:33.382149 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.382128 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-43.ec2.internal\" not found" node="ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.386501 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.386480 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-43.ec2.internal\" not found" node="ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.442666 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.442638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5888a8091ffcc6604df7d98fe068b1ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal\" (UID: \"5888a8091ffcc6604df7d98fe068b1ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.442761 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.442682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5888a8091ffcc6604df7d98fe068b1ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal\" (UID: \"5888a8091ffcc6604df7d98fe068b1ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.442761 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.442706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eec5dd9f88d629eb8d902b14a241ffc9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-43.ec2.internal\" (UID: \"eec5dd9f88d629eb8d902b14a241ffc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.451528 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.451508 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.543528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.543497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eec5dd9f88d629eb8d902b14a241ffc9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-43.ec2.internal\" (UID: \"eec5dd9f88d629eb8d902b14a241ffc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.543606 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.543532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5888a8091ffcc6604df7d98fe068b1ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal\" (UID: \"5888a8091ffcc6604df7d98fe068b1ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.543606 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.543566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5888a8091ffcc6604df7d98fe068b1ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal\" (UID: \"5888a8091ffcc6604df7d98fe068b1ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.543670 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.543609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eec5dd9f88d629eb8d902b14a241ffc9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-43.ec2.internal\" (UID: \"eec5dd9f88d629eb8d902b14a241ffc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.543702 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.543611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5888a8091ffcc6604df7d98fe068b1ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal\" (UID: \"5888a8091ffcc6604df7d98fe068b1ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.543702 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.543609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5888a8091ffcc6604df7d98fe068b1ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal\" (UID: \"5888a8091ffcc6604df7d98fe068b1ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.552594 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.552571 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.653468 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.653390 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.684591 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.684553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.689755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:33.689390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" Sep 10 19:10:33.754328 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.754282 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.854884 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.854863 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:33.955432 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:33.955355 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:34.055888 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:34.055865 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:34.077052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.077025 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 19:10:34.077208 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.077191 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 19:10:34.111448 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.111412 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:10:34.156763 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:34.156745 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:34.184918 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.184887 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 19:05:34 +0000 UTC" deadline="2028-04-21 08:48:43.954632309 +0000 UTC" Sep 10 19:10:34.184918 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.184916 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14125h38m9.769718632s" Sep 10 19:10:34.257314 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:34.257254 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-43.ec2.internal\" not found" Sep 10 19:10:34.277558 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.277398 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:10:34.278375 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.278357 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:10:34.278459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.278388 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:10:34.278459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.278404 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:10:34.352122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.352092 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:10:34.440953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.440924 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" Sep 10 19:10:34.460526 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.460507 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 19:10:34.462053 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:34.462029 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec5dd9f88d629eb8d902b14a241ffc9.slice/crio-2345342d1018991b182e8dd13846691154ebbbbe89c5e1c3f353b86e060a65d6 WatchSource:0}: Error finding container 2345342d1018991b182e8dd13846691154ebbbbe89c5e1c3f353b86e060a65d6: Status 404 returned error can't find the container with id 2345342d1018991b182e8dd13846691154ebbbbe89c5e1c3f353b86e060a65d6 Sep 10 19:10:34.462510 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:34.462487 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5888a8091ffcc6604df7d98fe068b1ea.slice/crio-ebf88662773c594b18e6bdf28ff0ef5e3ce8baa887a0225787d987c2e0420c27 WatchSource:0}: Error finding container ebf88662773c594b18e6bdf28ff0ef5e3ce8baa887a0225787d987c2e0420c27: Status 404 returned error can't find the container with id ebf88662773c594b18e6bdf28ff0ef5e3ce8baa887a0225787d987c2e0420c27 Sep 10 19:10:34.462560 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.462547 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" Sep 10 19:10:34.466367 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.466352 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:10:34.473883 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.473867 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 19:10:34.648702 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:34.648619 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:10:36.466853 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.466823 2576 apiserver.go:52] "Watching apiserver" Sep 10 19:10:36.474950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.474924 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 19:10:36.477431 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.477391 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal","openshift-multus/multus-additional-cni-plugins-dnd8d","openshift-multus/multus-j772x","openshift-network-diagnostics/network-check-target-4xrwb","openshift-network-operator/iptables-alerter-wjvlw","openshift-ovn-kubernetes/ovnkube-node-l2lfg","kube-system/global-pull-secret-syncer-qxlb7","kube-system/konnectivity-agent-f6t55","kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz","openshift-cluster-node-tuning-operator/tuned-h8hwb","openshift-dns/node-resolver-6qjq4","openshift-image-registry/node-ca-2wk66","openshift-multus/network-metrics-daemon-sr4z7"] Sep 10 19:10:36.480463 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.479878 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.482022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.482000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.482953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.482702 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 19:10:36.482953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.482929 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 19:10:36.483105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.482955 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz88g\"" Sep 10 19:10:36.483105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.482937 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 19:10:36.485398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.484243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j772x" Sep 10 19:10:36.485398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.484365 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 19:10:36.485398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.484675 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:10:36.485398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.484990 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg4jk\"" Sep 10 19:10:36.486447 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.486413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:36.486530 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.486450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 19:10:36.486530 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.486506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:36.486843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.486826 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 19:10:36.486843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.486835 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-44sd9\"" Sep 10 19:10:36.486960 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.486845 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 19:10:36.487812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.487099 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 19:10:36.488824 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.488804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:36.488917 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.488877 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:36.490424 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.490406 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 19:10:36.492649 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.492619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.494752 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.494736 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 19:10:36.495055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.495043 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 19:10:36.495183 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.495172 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f85vw\"" Sep 10 19:10:36.495325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.495315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.497657 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.497090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.497657 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.497563 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 19:10:36.497657 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.497584 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n2w68\"" Sep 10 19:10:36.497847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.497702 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 19:10:36.499398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.499357 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 19:10:36.499520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.499490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c7k4l\"" Sep 10 19:10:36.499520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.499499 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 19:10:36.499652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.499595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 19:10:36.500309 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.500292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.500872 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.500729 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 19:10:36.502516 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-daemon-config\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.502604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502542 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 19:10:36.502604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysctl-conf\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.502604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmn4\" (UniqueName: \"kubernetes.io/projected/7ba2472f-ced7-450b-bdbc-24da03cab607-kube-api-access-qgmn4\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-run\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-system-cni-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-cni-binary-copy\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-k8s-cni-cncf-io\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-socket-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.502754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-registration-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.502771 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysconfig\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-netns\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-sys\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f69e4390-2d3a-482f-b100-fabc3d646799-tmp\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdjl\" (UniqueName: \"kubernetes.io/projected/f69e4390-2d3a-482f-b100-fabc3d646799-kube-api-access-vsdjl\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33146da8-d5b7-4b63-aebf-2c12ae94fb5a-konnectivity-ca\") pod \"konnectivity-agent-f6t55\" (UID: \"33146da8-d5b7-4b63-aebf-2c12ae94fb5a\") " pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-device-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-sys-fs\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.502994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-modprobe-d\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-systemd\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f69e4390-2d3a-482f-b100-fabc3d646799-etc-tuned\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-kubernetes\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-cni-bin\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-hostroot\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-multus-certs\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-etc-kubernetes\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysctl-d\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-host\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-cni-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-cni-multus\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxj4d\" (UniqueName: \"kubernetes.io/projected/04d5421b-2ce5-469d-98e5-9fc2059de454-kube-api-access-wxj4d\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-cnibin\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-socket-dir-parent\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-kubelet\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmp6\" (UniqueName: \"kubernetes.io/projected/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-kube-api-access-tzmp6\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-lib-modules\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-var-lib-kubelet\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33146da8-d5b7-4b63-aebf-2c12ae94fb5a-agent-certs\") pod \"konnectivity-agent-f6t55\" (UID: \"33146da8-d5b7-4b63-aebf-2c12ae94fb5a\") " pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.503909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-conf-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.504547 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503711 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 19:10:36.504547 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.503739 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ckj79\"" Sep 10 19:10:36.504713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.504698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.507094 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.507078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.510318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.509128 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 19:10:36.510318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.509258 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 19:10:36.510318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.509388 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 19:10:36.510318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.509475 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5zgxn\"" Sep 10 19:10:36.510318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.509734 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 19:10:36.510318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.509991 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 19:10:36.510655 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.510506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 19:10:36.510752 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.510730 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 19:10:36.510859 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.510768 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:10:36.510859 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.510732 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gwx4d\"" Sep 10 19:10:36.511005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.510901 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 19:10:36.524531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.524277 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-79ws6" Sep 10 19:10:36.534400 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.534363 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-79ws6" Sep 10 19:10:36.592154 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.592123 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 19:10:36.602639 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.602590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" event={"ID":"eec5dd9f88d629eb8d902b14a241ffc9","Type":"ContainerStarted","Data":"2345342d1018991b182e8dd13846691154ebbbbe89c5e1c3f353b86e060a65d6"} Sep 10 19:10:36.603703 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovnkube-config\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.603811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.603811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-systemd\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.603811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f69e4390-2d3a-482f-b100-fabc3d646799-etc-tuned\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.603811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-run-netns\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.603811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-hosts-file\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-host\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-cni-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-cni-multus\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-systemd-units\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-cni-netd\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxj4d\" (UniqueName: \"kubernetes.io/projected/04d5421b-2ce5-469d-98e5-9fc2059de454-kube-api-access-wxj4d\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-cnibin\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.603990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-lib-modules\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-conf-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.604104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsfc\" (UniqueName: \"kubernetes.io/projected/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-kube-api-access-qhsfc\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-host\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-systemd\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-var-lib-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-cnibin\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysctl-conf\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pvc\" (UniqueName: \"kubernetes.io/projected/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-kube-api-access-85pvc\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-systemd\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-kubelet\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-log-socket\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovnkube-script-lib\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmn4\" (UniqueName: \"kubernetes.io/projected/7ba2472f-ced7-450b-bdbc-24da03cab607-kube-api-access-qgmn4\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-run\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" event={"ID":"5888a8091ffcc6604df7d98fe068b1ea","Type":"ContainerStarted","Data":"ebf88662773c594b18e6bdf28ff0ef5e3ce8baa887a0225787d987c2e0420c27"} Sep 10 19:10:36.604543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-k8s-cni-cncf-io\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-k8s-cni-cncf-io\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-etc-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcf9q\" (UniqueName: \"kubernetes.io/projected/7cdc9edd-7db3-4019-a574-40120bfc42fd-kube-api-access-wcf9q\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-registration-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-netns\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-cni-bin\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604615 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604642 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-lib-modules\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-sys\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f69e4390-2d3a-482f-b100-fabc3d646799-tmp\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-device-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-modprobe-d\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-kubernetes\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-cni-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b14c0d55-d660-472c-ad93-f42bf3893fe9-dbus\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-node-log\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-conf-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-netns\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-cni-bin\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysctl-conf\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-etc-kubernetes\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.604995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-ovn\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-sys\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovn-node-metrics-cert\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysctl-d\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-socket-dir-parent\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-kubelet\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmp6\" (UniqueName: \"kubernetes.io/projected/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-kube-api-access-tzmp6\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8llv6\" (UniqueName: \"kubernetes.io/projected/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-kube-api-access-8llv6\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.605548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-var-lib-kubelet\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33146da8-d5b7-4b63-aebf-2c12ae94fb5a-agent-certs\") pod \"konnectivity-agent-f6t55\" (UID: \"33146da8-d5b7-4b63-aebf-2c12ae94fb5a\") " pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-daemon-config\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-tmp-dir\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-host\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-serviceca\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-run\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-iptables-alerter-script\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-system-cni-dir\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-env-overrides\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-system-cni-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-cni-binary-copy\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-host-slash\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-cni-multus\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-slash\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92p7t\" (UniqueName: \"kubernetes.io/projected/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-kube-api-access-92p7t\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.606349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-device-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-socket-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysconfig\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b14c0d55-d660-472c-ad93-f42bf3893fe9-kubelet-config\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cnibin\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdjl\" (UniqueName: \"kubernetes.io/projected/f69e4390-2d3a-482f-b100-fabc3d646799-kube-api-access-vsdjl\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33146da8-d5b7-4b63-aebf-2c12ae94fb5a-konnectivity-ca\") pod \"konnectivity-agent-f6t55\" (UID: \"33146da8-d5b7-4b63-aebf-2c12ae94fb5a\") " pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-sys-fs\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-hostroot\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-multus-certs\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-cni-binary-copy\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-run-multus-certs\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-modprobe-d\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-kubelet\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-registration-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-sys-fs\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.607215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-host-var-lib-cni-bin\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-etc-kubernetes\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606104 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysctl-d\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606151 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-socket-dir-parent\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-hostroot\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-multus-daemon-config\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.606480 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.605847 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-kubernetes\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606552 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ba2472f-ced7-450b-bdbc-24da03cab607-socket-dir\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.606566 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs podName:04d5421b-2ce5-469d-98e5-9fc2059de454 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:37.106547506 +0000 UTC m=+3.023203533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs") pod "network-metrics-daemon-sr4z7" (UID: "04d5421b-2ce5-469d-98e5-9fc2059de454") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33146da8-d5b7-4b63-aebf-2c12ae94fb5a-konnectivity-ca\") pod \"konnectivity-agent-f6t55\" (UID: \"33146da8-d5b7-4b63-aebf-2c12ae94fb5a\") " pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-system-cni-dir\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606833 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-etc-sysconfig\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.607886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.606853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f69e4390-2d3a-482f-b100-fabc3d646799-var-lib-kubelet\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.608320 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.608086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f69e4390-2d3a-482f-b100-fabc3d646799-tmp\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.608320 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.608210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f69e4390-2d3a-482f-b100-fabc3d646799-etc-tuned\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.608873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.608850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33146da8-d5b7-4b63-aebf-2c12ae94fb5a-agent-certs\") pod \"konnectivity-agent-f6t55\" (UID: \"33146da8-d5b7-4b63-aebf-2c12ae94fb5a\") " pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.621238 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.621177 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:10:36.621238 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.621203 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:10:36.621238 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.621217 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jzqqs for pod openshift-network-diagnostics/network-check-target-4xrwb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:36.621450 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.621283 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs podName:6a2c4140-61fe-483c-b71f-3d802b27a08f nodeName:}" failed. No retries permitted until 2026-09-10 19:10:37.121265849 +0000 UTC m=+3.037921876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jzqqs" (UniqueName: "kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs") pod "network-check-target-4xrwb" (UID: "6a2c4140-61fe-483c-b71f-3d802b27a08f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:36.622043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.622026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmn4\" (UniqueName: \"kubernetes.io/projected/7ba2472f-ced7-450b-bdbc-24da03cab607-kube-api-access-qgmn4\") pod \"aws-ebs-csi-driver-node-pnpmz\" (UID: \"7ba2472f-ced7-450b-bdbc-24da03cab607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.626429 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.626405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmp6\" (UniqueName: \"kubernetes.io/projected/96ce8c1a-27ff-42b2-9b27-621c1cdc4203-kube-api-access-tzmp6\") pod \"multus-j772x\" (UID: \"96ce8c1a-27ff-42b2-9b27-621c1cdc4203\") " pod="openshift-multus/multus-j772x" Sep 10 19:10:36.627419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.627398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdjl\" (UniqueName: \"kubernetes.io/projected/f69e4390-2d3a-482f-b100-fabc3d646799-kube-api-access-vsdjl\") pod \"tuned-h8hwb\" (UID: \"f69e4390-2d3a-482f-b100-fabc3d646799\") " pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.628733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.628711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxj4d\" (UniqueName: \"kubernetes.io/projected/04d5421b-2ce5-469d-98e5-9fc2059de454-kube-api-access-wxj4d\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:36.709070 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-env-overrides\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709241 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-host-slash\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.709241 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-slash\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709241 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92p7t\" (UniqueName: \"kubernetes.io/projected/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-kube-api-access-92p7t\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.709241 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b14c0d55-d660-472c-ad93-f42bf3893fe9-kubelet-config\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.709462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cnibin\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.709462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovnkube-config\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709561 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-run-netns\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709561 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-hosts-file\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.709561 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.709736 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-systemd-units\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709795 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-host-slash\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.709795 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-cni-netd\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709795 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-slash\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.709906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-run-netns\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.709906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsfc\" (UniqueName: \"kubernetes.io/projected/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-kube-api-access-qhsfc\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.710026 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.710026 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.709969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-systemd\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710026 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-var-lib-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710026 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pvc\" (UniqueName: \"kubernetes.io/projected/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-kube-api-access-85pvc\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.710132 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.710026 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:36.710132 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.710132 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-kubelet\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710132 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-log-socket\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710132 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovnkube-script-lib\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710132 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:36.710103 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret podName:b14c0d55-d660-472c-ad93-f42bf3893fe9 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:37.210088932 +0000 UTC m=+3.126744938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret") pod "global-pull-secret-syncer-qxlb7" (UID: "b14c0d55-d660-472c-ad93-f42bf3893fe9") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:36.710338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.710338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-etc-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcf9q\" (UniqueName: \"kubernetes.io/projected/7cdc9edd-7db3-4019-a574-40120bfc42fd-kube-api-access-wcf9q\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-cni-bin\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b14c0d55-d660-472c-ad93-f42bf3893fe9-dbus\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.710563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-node-log\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-ovn\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovn-node-metrics-cert\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8llv6\" (UniqueName: \"kubernetes.io/projected/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-kube-api-access-8llv6\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.710563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-tmp-dir\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-host\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-serviceca\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-iptables-alerter-script\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-hosts-file\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-system-cni-dir\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovnkube-script-lib\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-system-cni-dir\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.710840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cnibin\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.711285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-systemd-units\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b14c0d55-d660-472c-ad93-f42bf3893fe9-dbus\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.711285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.710948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-cni-netd\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-systemd\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-var-lib-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovnkube-config\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711447 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-node-log\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.711634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-run-ovn\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-kubelet\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711895 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-log-socket\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711942 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-etc-openvswitch\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.711997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-host\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.712046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.712496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-tmp-dir\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.712588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-iptables-alerter-script\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.712659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b14c0d55-d660-472c-ad93-f42bf3893fe9-kubelet-config\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.712712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cdc9edd-7db3-4019-a574-40120bfc42fd-host-cni-bin\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.712782 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-serviceca\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.713587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.713056 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cdc9edd-7db3-4019-a574-40120bfc42fd-env-overrides\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.714212 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.713970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.716754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.716735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cdc9edd-7db3-4019-a574-40120bfc42fd-ovn-node-metrics-cert\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.719204 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.719184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92p7t\" (UniqueName: \"kubernetes.io/projected/0f6c2cf3-35b5-4054-8b11-58f48bc41faf-kube-api-access-92p7t\") pod \"node-resolver-6qjq4\" (UID: \"0f6c2cf3-35b5-4054-8b11-58f48bc41faf\") " pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.721861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.721842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pvc\" (UniqueName: \"kubernetes.io/projected/e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd-kube-api-access-85pvc\") pod \"node-ca-2wk66\" (UID: \"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd\") " pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.721950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.721893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsfc\" (UniqueName: \"kubernetes.io/projected/58ba63e0-b0b4-4827-bd6b-20f4d1c49691-kube-api-access-qhsfc\") pod \"iptables-alerter-wjvlw\" (UID: \"58ba63e0-b0b4-4827-bd6b-20f4d1c49691\") " pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.722639 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.722603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcf9q\" (UniqueName: \"kubernetes.io/projected/7cdc9edd-7db3-4019-a574-40120bfc42fd-kube-api-access-wcf9q\") pod \"ovnkube-node-l2lfg\" (UID: \"7cdc9edd-7db3-4019-a574-40120bfc42fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.727635 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.727607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8llv6\" (UniqueName: \"kubernetes.io/projected/7d3d0fc0-11c5-4f7e-add8-707f5640cb28-kube-api-access-8llv6\") pod \"multus-additional-cni-plugins-dnd8d\" (UID: \"7d3d0fc0-11c5-4f7e-add8-707f5640cb28\") " pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.793471 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.793436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" Sep 10 19:10:36.801214 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.801113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" Sep 10 19:10:36.810871 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:36.810841 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69e4390_2d3a_482f_b100_fabc3d646799.slice/crio-827730c05de1b0d0e74c6e6932306687772af0b2e8c879286cf38a2405dd38ed WatchSource:0}: Error finding container 827730c05de1b0d0e74c6e6932306687772af0b2e8c879286cf38a2405dd38ed: Status 404 returned error can't find the container with id 827730c05de1b0d0e74c6e6932306687772af0b2e8c879286cf38a2405dd38ed Sep 10 19:10:36.812239 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.812187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j772x" Sep 10 19:10:36.818112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.817950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:10:36.821618 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:36.821587 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ce8c1a_27ff_42b2_9b27_621c1cdc4203.slice/crio-c3df419dd24a259b13eaace7364964f3a43116a15d9a3bf84a3e4faaed2c4be0 WatchSource:0}: Error finding container c3df419dd24a259b13eaace7364964f3a43116a15d9a3bf84a3e4faaed2c4be0: Status 404 returned error can't find the container with id c3df419dd24a259b13eaace7364964f3a43116a15d9a3bf84a3e4faaed2c4be0 Sep 10 19:10:36.827358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.827334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6qjq4" Sep 10 19:10:36.834311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.834112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2wk66" Sep 10 19:10:36.838300 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:36.838233 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6c2cf3_35b5_4054_8b11_58f48bc41faf.slice/crio-1df75cd51c30d4fc7351b9372276320e7a495c5142a34ffce4fe843d63163aef WatchSource:0}: Error finding container 1df75cd51c30d4fc7351b9372276320e7a495c5142a34ffce4fe843d63163aef: Status 404 returned error can't find the container with id 1df75cd51c30d4fc7351b9372276320e7a495c5142a34ffce4fe843d63163aef Sep 10 19:10:36.841475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.841456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" Sep 10 19:10:36.843977 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:36.843863 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d7c0b_9cbd_4e83_ab81_b24c51c1b0bd.slice/crio-8eb8e6a0b7e90a639ea1ce5e3500e5e2fbd4c2095c6d267915efc51803a9d9b7 WatchSource:0}: Error finding container 8eb8e6a0b7e90a639ea1ce5e3500e5e2fbd4c2095c6d267915efc51803a9d9b7: Status 404 returned error can't find the container with id 8eb8e6a0b7e90a639ea1ce5e3500e5e2fbd4c2095c6d267915efc51803a9d9b7 Sep 10 19:10:36.849829 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.849810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:10:36.857115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.857095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjvlw" Sep 10 19:10:36.973913 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:36.973832 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:10:37.112845 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.112806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:37.113025 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.112962 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:37.113025 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.113024 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs podName:04d5421b-2ce5-469d-98e5-9fc2059de454 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:38.113003948 +0000 UTC m=+4.029659974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs") pod "network-metrics-daemon-sr4z7" (UID: "04d5421b-2ce5-469d-98e5-9fc2059de454") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:37.213425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.213395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:37.213425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.213443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:37.213624 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.213544 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:10:37.213624 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.213559 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:10:37.213624 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.213568 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jzqqs for pod openshift-network-diagnostics/network-check-target-4xrwb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:37.213624 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.213544 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:37.213624 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.213610 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs podName:6a2c4140-61fe-483c-b71f-3d802b27a08f nodeName:}" failed. No retries permitted until 2026-09-10 19:10:38.213598215 +0000 UTC m=+4.130254247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzqqs" (UniqueName: "kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs") pod "network-check-target-4xrwb" (UID: "6a2c4140-61fe-483c-b71f-3d802b27a08f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:37.213916 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.213646 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret podName:b14c0d55-d660-472c-ad93-f42bf3893fe9 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:38.213630764 +0000 UTC m=+4.130286771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret") pod "global-pull-secret-syncer-qxlb7" (UID: "b14c0d55-d660-472c-ad93-f42bf3893fe9") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:37.537361 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.537329 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:05:36 +0000 UTC" deadline="2028-04-21 11:33:52.156766123 +0000 UTC" Sep 10 19:10:37.537361 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.537356 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14128h23m14.619412617s" Sep 10 19:10:37.596037 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.596007 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:37.596172 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:37.596152 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:37.607925 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.607897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j772x" event={"ID":"96ce8c1a-27ff-42b2-9b27-621c1cdc4203","Type":"ContainerStarted","Data":"c3df419dd24a259b13eaace7364964f3a43116a15d9a3bf84a3e4faaed2c4be0"} Sep 10 19:10:37.609242 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.609215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" event={"ID":"f69e4390-2d3a-482f-b100-fabc3d646799","Type":"ContainerStarted","Data":"827730c05de1b0d0e74c6e6932306687772af0b2e8c879286cf38a2405dd38ed"} Sep 10 19:10:37.610502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.610474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" event={"ID":"7ba2472f-ced7-450b-bdbc-24da03cab607","Type":"ContainerStarted","Data":"927eb5183584a6183e3fb89b212bbc2e97236253d1f71647768dbaae1e510c52"} Sep 10 19:10:37.611679 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.611642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2wk66" event={"ID":"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd","Type":"ContainerStarted","Data":"8eb8e6a0b7e90a639ea1ce5e3500e5e2fbd4c2095c6d267915efc51803a9d9b7"} Sep 10 19:10:37.612868 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.612843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6qjq4" event={"ID":"0f6c2cf3-35b5-4054-8b11-58f48bc41faf","Type":"ContainerStarted","Data":"1df75cd51c30d4fc7351b9372276320e7a495c5142a34ffce4fe843d63163aef"} Sep 10 19:10:37.613992 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:37.613973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6t55" event={"ID":"33146da8-d5b7-4b63-aebf-2c12ae94fb5a","Type":"ContainerStarted","Data":"6759031a275f4e677dad385e618b0346a349e64c6e23d8f5e8c18b59c11229ac"} Sep 10 19:10:37.889618 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:37.889576 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ba63e0_b0b4_4827_bd6b_20f4d1c49691.slice/crio-3f5d84a704cff01de9cf445d60b2d67e765aec0d12e416232de88f9760013646 WatchSource:0}: Error finding container 3f5d84a704cff01de9cf445d60b2d67e765aec0d12e416232de88f9760013646: Status 404 returned error can't find the container with id 3f5d84a704cff01de9cf445d60b2d67e765aec0d12e416232de88f9760013646 Sep 10 19:10:37.891051 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:10:37.891008 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3d0fc0_11c5_4f7e_add8_707f5640cb28.slice/crio-d30dc96150ac94c748e7634bdc70786919d636dfc1da1d10a0a9cacc5b98af9d WatchSource:0}: Error finding container d30dc96150ac94c748e7634bdc70786919d636dfc1da1d10a0a9cacc5b98af9d: Status 404 returned error can't find the container with id d30dc96150ac94c748e7634bdc70786919d636dfc1da1d10a0a9cacc5b98af9d Sep 10 19:10:38.119298 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.119212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:38.119488 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.119345 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:38.119488 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.119441 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs podName:04d5421b-2ce5-469d-98e5-9fc2059de454 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:40.119423368 +0000 UTC m=+6.036079376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs") pod "network-metrics-daemon-sr4z7" (UID: "04d5421b-2ce5-469d-98e5-9fc2059de454") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.219680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.219746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.219909 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.219931 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.219943 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jzqqs for pod openshift-network-diagnostics/network-check-target-4xrwb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.219998 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs podName:6a2c4140-61fe-483c-b71f-3d802b27a08f nodeName:}" failed. No retries permitted until 2026-09-10 19:10:40.219978999 +0000 UTC m=+6.136635020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzqqs" (UniqueName: "kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs") pod "network-check-target-4xrwb" (UID: "6a2c4140-61fe-483c-b71f-3d802b27a08f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.220328 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:38.220443 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.220365 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret podName:b14c0d55-d660-472c-ad93-f42bf3893fe9 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:40.220355762 +0000 UTC m=+6.137011768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret") pod "global-pull-secret-syncer-qxlb7" (UID: "b14c0d55-d660-472c-ad93-f42bf3893fe9") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:38.538621 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.538529 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:05:36 +0000 UTC" deadline="2028-06-03 14:41:06.295054978 +0000 UTC" Sep 10 19:10:38.538621 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.538574 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15163h30m27.756486145s" Sep 10 19:10:38.596886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.596640 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:38.596886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.596862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:38.597054 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.596997 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:38.597471 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:38.597445 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:38.619797 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.619765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"c12a333a2b49c8fac5684a0dd310b5a4d7e1ec1cc219ae638973b51aa8a5e5e3"} Sep 10 19:10:38.624347 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.624315 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerStarted","Data":"d30dc96150ac94c748e7634bdc70786919d636dfc1da1d10a0a9cacc5b98af9d"} Sep 10 19:10:38.632054 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.632027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjvlw" event={"ID":"58ba63e0-b0b4-4827-bd6b-20f4d1c49691","Type":"ContainerStarted","Data":"3f5d84a704cff01de9cf445d60b2d67e765aec0d12e416232de88f9760013646"} Sep 10 19:10:38.637654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.637626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" event={"ID":"eec5dd9f88d629eb8d902b14a241ffc9","Type":"ContainerStarted","Data":"53015107aa763e373f170ce70f6e07dd1687ba88263bd9924392da763d625bf3"} Sep 10 19:10:38.643363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.642924 2576 generic.go:356] "Generic (PLEG): container finished" podID="5888a8091ffcc6604df7d98fe068b1ea" containerID="bd18889bbc3b2730a36d19d34577023082e7d7e2265b87e660fbe4a14cee9e5f" exitCode=0 Sep 10 19:10:38.643363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.642958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" event={"ID":"5888a8091ffcc6604df7d98fe068b1ea","Type":"ContainerDied","Data":"bd18889bbc3b2730a36d19d34577023082e7d7e2265b87e660fbe4a14cee9e5f"} Sep 10 19:10:38.694345 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:38.693976 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-43.ec2.internal" podStartSLOduration=3.693953874 podStartE2EDuration="3.693953874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:38.665032396 +0000 UTC m=+4.581688426" watchObservedRunningTime="2026-09-10 19:10:38.693953874 +0000 UTC m=+4.610609906" Sep 10 19:10:39.596226 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:39.596189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:39.596713 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:39.596317 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:39.664518 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:39.664438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" event={"ID":"5888a8091ffcc6604df7d98fe068b1ea","Type":"ContainerStarted","Data":"e6cd5baed846ea0318f0c4f232adb4255478f4112f826a5a0990b436c4f2afce"} Sep 10 19:10:40.140120 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:40.140076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:40.140292 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.140278 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:40.140386 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.140344 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs podName:04d5421b-2ce5-469d-98e5-9fc2059de454 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:44.140323926 +0000 UTC m=+10.056979935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs") pod "network-metrics-daemon-sr4z7" (UID: "04d5421b-2ce5-469d-98e5-9fc2059de454") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:40.241594 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:40.241400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:40.241594 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:40.241490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:40.241594 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.241601 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:40.241879 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.241660 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret podName:b14c0d55-d660-472c-ad93-f42bf3893fe9 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:44.241642842 +0000 UTC m=+10.158298855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret") pod "global-pull-secret-syncer-qxlb7" (UID: "b14c0d55-d660-472c-ad93-f42bf3893fe9") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:40.242077 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.242057 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:10:40.242138 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.242083 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:10:40.242138 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.242096 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jzqqs for pod openshift-network-diagnostics/network-check-target-4xrwb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:40.242243 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.242141 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs podName:6a2c4140-61fe-483c-b71f-3d802b27a08f nodeName:}" failed. No retries permitted until 2026-09-10 19:10:44.242125609 +0000 UTC m=+10.158781622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzqqs" (UniqueName: "kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs") pod "network-check-target-4xrwb" (UID: "6a2c4140-61fe-483c-b71f-3d802b27a08f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:40.597583 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:40.597502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:40.598026 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.597648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:40.599614 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:40.599592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:40.599735 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:40.599709 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:41.596332 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:41.596299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:41.596532 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:41.596421 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:42.597361 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:42.597331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:42.597813 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:42.597471 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:42.597813 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:42.597542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:42.597813 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:42.597651 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:43.596011 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:43.595932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:43.596133 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:43.596058 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:44.174550 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:44.174515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:44.174999 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.174696 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:44.174999 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.174779 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs podName:04d5421b-2ce5-469d-98e5-9fc2059de454 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:52.174756634 +0000 UTC m=+18.091412643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs") pod "network-metrics-daemon-sr4z7" (UID: "04d5421b-2ce5-469d-98e5-9fc2059de454") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:44.275399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:44.275492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.275583 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.275609 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.275610 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.275622 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jzqqs for pod openshift-network-diagnostics/network-check-target-4xrwb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.275678 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs podName:6a2c4140-61fe-483c-b71f-3d802b27a08f nodeName:}" failed. No retries permitted until 2026-09-10 19:10:52.275657806 +0000 UTC m=+18.192313831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzqqs" (UniqueName: "kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs") pod "network-check-target-4xrwb" (UID: "6a2c4140-61fe-483c-b71f-3d802b27a08f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:44.275683 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.275698 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret podName:b14c0d55-d660-472c-ad93-f42bf3893fe9 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:52.275688863 +0000 UTC m=+18.192344875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret") pod "global-pull-secret-syncer-qxlb7" (UID: "b14c0d55-d660-472c-ad93-f42bf3893fe9") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:44.596080 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:44.596005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:44.596231 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.596139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:44.597215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:44.597193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:44.597309 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:44.597265 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:45.595930 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:45.595893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:45.596345 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:45.596027 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:46.600267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:46.600233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:46.600757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:46.600245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:46.600757 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:46.600367 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:46.600757 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:46.600458 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:47.596806 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:47.596265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:47.596806 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:47.596421 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:48.596671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:48.596631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:48.597000 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:48.596678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:48.597000 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:48.596757 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:48.597000 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:48.596878 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:49.596350 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:49.596306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:49.596574 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:49.596460 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:50.599258 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:50.599229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:50.599699 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:50.599387 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:50.599699 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:50.599427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:50.599699 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:50.599542 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:51.596392 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:51.596341 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:51.596589 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:51.596478 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:52.231836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:52.231806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:52.232236 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.231958 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:52.232236 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.232027 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs podName:04d5421b-2ce5-469d-98e5-9fc2059de454 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:08.232010228 +0000 UTC m=+34.148666240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs") pod "network-metrics-daemon-sr4z7" (UID: "04d5421b-2ce5-469d-98e5-9fc2059de454") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:10:52.332318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:52.332284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:52.332467 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:52.332329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:52.332467 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.332444 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:52.332467 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.332454 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:10:52.332467 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.332469 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:10:52.332622 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.332478 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jzqqs for pod openshift-network-diagnostics/network-check-target-4xrwb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:52.332622 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.332511 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret podName:b14c0d55-d660-472c-ad93-f42bf3893fe9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:08.332494818 +0000 UTC m=+34.249150826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret") pod "global-pull-secret-syncer-qxlb7" (UID: "b14c0d55-d660-472c-ad93-f42bf3893fe9") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:10:52.332622 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.332524 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs podName:6a2c4140-61fe-483c-b71f-3d802b27a08f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:08.332518709 +0000 UTC m=+34.249174716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzqqs" (UniqueName: "kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs") pod "network-check-target-4xrwb" (UID: "6a2c4140-61fe-483c-b71f-3d802b27a08f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:10:52.599300 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:52.599230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:52.599300 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:52.599266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:52.599512 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.599327 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:52.599512 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:52.599479 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:53.596579 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:53.596546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:53.596973 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:53.596681 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:54.597350 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:54.597317 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:54.597727 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:54.597470 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:54.597727 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:54.597517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:54.597727 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:54.597611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:55.595831 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:55.595786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:55.596000 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:55.595944 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:56.596572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.596332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:56.597222 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.596412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:56.597222 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:56.596667 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:56.597222 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:56.596723 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:56.702564 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.702530 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2wk66" event={"ID":"e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd","Type":"ContainerStarted","Data":"528cebc04f362c0a1a89d7cadd41aacd1a953e9c521d9adb5d3618d9c5a8c7be"} Sep 10 19:10:56.704134 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.704104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6qjq4" event={"ID":"0f6c2cf3-35b5-4054-8b11-58f48bc41faf","Type":"ContainerStarted","Data":"a9b38e78ed281d2d034de3e643344f2d55808426f8a5e46884822d3c72e62953"} Sep 10 19:10:56.705641 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.705615 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6t55" event={"ID":"33146da8-d5b7-4b63-aebf-2c12ae94fb5a","Type":"ContainerStarted","Data":"4da306b96bb69197bc72fd63dd39354831ee6cc468c6120e51bda04f12639dc0"} Sep 10 19:10:56.707164 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.707144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j772x" event={"ID":"96ce8c1a-27ff-42b2-9b27-621c1cdc4203","Type":"ContainerStarted","Data":"0f600a247f9200add1c216f0834f4117f5bd99f883f8cc3e74bd4aca818cdaee"} Sep 10 19:10:56.708505 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.708485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" event={"ID":"7ba2472f-ced7-450b-bdbc-24da03cab607","Type":"ContainerStarted","Data":"286b93daf82c8bd321a644a236145a1a0c60a00ea04ca7db5efcaabf01e19ace"} Sep 10 19:10:56.710343 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.710326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"7ffedf9b83cd45c37c872f37868696364bdd078b726f4d9a526d29b0dad8cb1e"} Sep 10 19:10:56.710444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.710407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"7198ba62095c7e572ccbc1c546baa06db36d2a00222d3b36524219191c12ba61"} Sep 10 19:10:56.711835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.711813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" event={"ID":"f69e4390-2d3a-482f-b100-fabc3d646799","Type":"ContainerStarted","Data":"dec1a0ae6c98da050858650c7bd794e1267fa0f8e4cf8bebd7f21597f7817ca7"} Sep 10 19:10:56.713404 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.713364 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d3d0fc0-11c5-4f7e-add8-707f5640cb28" containerID="bffa9bc4b928369e383003d23d41acebe0a5b6c4b6e88f25fb390626eeaa1cb3" exitCode=0 Sep 10 19:10:56.713491 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.713406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerDied","Data":"bffa9bc4b928369e383003d23d41acebe0a5b6c4b6e88f25fb390626eeaa1cb3"} Sep 10 19:10:56.727768 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.727621 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6qjq4" podStartSLOduration=3.406148334 podStartE2EDuration="22.727609498s" totalImagesPullingTime="19.321461164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:56.727467313 +0000 UTC m=+22.644123342" watchObservedRunningTime="2026-09-10 19:10:56.727609498 +0000 UTC m=+22.644265526" Sep 10 19:10:56.727845 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.727830 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-43.ec2.internal" podStartSLOduration=21.727824415 podStartE2EDuration="21.727824415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:39.681720866 +0000 UTC m=+5.598376896" watchObservedRunningTime="2026-09-10 19:10:56.727824415 +0000 UTC m=+22.644480445" Sep 10 19:10:56.743635 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.743600 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2wk66" podStartSLOduration=3.83224666 podStartE2EDuration="22.743586829s" totalImagesPullingTime="18.911340169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:56.743444714 +0000 UTC m=+22.660100743" watchObservedRunningTime="2026-09-10 19:10:56.743586829 +0000 UTC m=+22.660242858" Sep 10 19:10:56.760903 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.760861 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f6t55" podStartSLOduration=3.430073316 podStartE2EDuration="22.760846926s" totalImagesPullingTime="19.33077361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:56.760761323 +0000 UTC m=+22.677417352" watchObservedRunningTime="2026-09-10 19:10:56.760846926 +0000 UTC m=+22.677502956" Sep 10 19:10:56.810001 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.809966 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j772x" podStartSLOduration=3.405446351 podStartE2EDuration="22.809951555s" totalImagesPullingTime="19.404505204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:56.809153332 +0000 UTC m=+22.725809362" watchObservedRunningTime="2026-09-10 19:10:56.809951555 +0000 UTC m=+22.726607584" Sep 10 19:10:56.830735 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:56.830698 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8hwb" podStartSLOduration=3.442705568 podStartE2EDuration="22.830684344s" totalImagesPullingTime="19.387978776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:56.830274025 +0000 UTC m=+22.746930051" watchObservedRunningTime="2026-09-10 19:10:56.830684344 +0000 UTC m=+22.747340372" Sep 10 19:10:57.595648 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.595617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:57.595901 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:57.595723 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:57.719703 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.719665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"637538184a9394cbf23cfd12dddc6793375277f411b033cc59fafcc8561ba620"} Sep 10 19:10:57.720259 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.719776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"e528db9bd6a0af140a147a5060e34c544272c7b5cece34bd49dde9b1b1c84365"} Sep 10 19:10:57.720259 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.719838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"59405525c14da80825e0034aa76eaf73a58d29c065e21e020fad49b658084e75"} Sep 10 19:10:57.720259 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.719882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"439298aa4727e2a1786ccc26662b4924f139ba0567ee516c91166876b3e57327"} Sep 10 19:10:57.721501 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.721469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjvlw" event={"ID":"58ba63e0-b0b4-4827-bd6b-20f4d1c49691","Type":"ContainerStarted","Data":"ff9d84ff022405da21b73c0b65131b947f4d1b37a6dea43c9895da90dfa1e238"} Sep 10 19:10:57.738007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.737914 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wjvlw" podStartSLOduration=4.428009657 podStartE2EDuration="22.737895453s" totalImagesPullingTime="18.309885796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:57.737632928 +0000 UTC m=+23.654288957" watchObservedRunningTime="2026-09-10 19:10:57.737895453 +0000 UTC m=+23.654551491" Sep 10 19:10:57.966916 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:57.966886 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 19:10:58.574265 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:58.574171 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T19:10:57.96691113Z","UUID":"df41b65b-3e9e-47e5-9554-6bcc68c1e21b","Handler":null,"Name":"","Endpoint":""} Sep 10 19:10:58.575829 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:58.575807 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 19:10:58.575909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:58.575842 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 19:10:58.596996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:58.595937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:10:58.596996 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:58.596140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:10:58.596996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:58.596952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:10:58.597339 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:58.597046 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:10:58.726339 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:58.726302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" event={"ID":"7ba2472f-ced7-450b-bdbc-24da03cab607","Type":"ContainerStarted","Data":"9068d7f62bc6a402cc819aa6ce0509c11d165d10c9d7b8a33b0d287deab1a725"} Sep 10 19:10:59.596158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:59.596089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:10:59.596291 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:10:59.596269 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:10:59.731052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:59.731018 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" event={"ID":"7ba2472f-ced7-450b-bdbc-24da03cab607","Type":"ContainerStarted","Data":"e6b26b66e78884f790566dcce5bb7b443e1eb27ab9b7cad2b9b246699621bf35"} Sep 10 19:10:59.734809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:59.734779 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"3a61d19a0f32e36105bfe0dfa75a7bd98fee4fe921aa5ce27277f0161e133062"} Sep 10 19:10:59.749881 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:10:59.749835 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnpmz" podStartSLOduration=3.474076748 podStartE2EDuration="25.749819977s" totalImagesPullingTime="22.275743229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:59.749098756 +0000 UTC m=+25.665754785" watchObservedRunningTime="2026-09-10 19:10:59.749819977 +0000 UTC m=+25.666476005" Sep 10 19:11:00.599692 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:00.599660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:00.599894 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:00.599808 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:11:00.600152 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:00.600132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:00.600238 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:00.600213 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:11:01.252366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:01.252183 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:11:01.253051 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:01.253029 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:11:01.596398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:01.596305 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:01.596536 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:01.596454 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:11:01.744772 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:01.744617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" event={"ID":"7cdc9edd-7db3-4019-a574-40120bfc42fd","Type":"ContainerStarted","Data":"ed23458f3db137c4ac9164a2ea8d05e7753b71fa6dc041145cae37d4ff7cc905"} Sep 10 19:11:01.745285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:01.745060 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:11:01.745779 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:01.745572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f6t55" Sep 10 19:11:01.777878 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:01.777833 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" podStartSLOduration=8.225528207 podStartE2EDuration="26.777817467s" totalImagesPullingTime="18.55228926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:01.777646135 +0000 UTC m=+27.694302178" watchObservedRunningTime="2026-09-10 19:11:01.777817467 +0000 UTC m=+27.694473495" Sep 10 19:11:02.599070 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.599041 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:02.599671 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:02.599144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:11:02.599671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.599198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:02.599671 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:02.599288 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:11:02.748413 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.748365 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d3d0fc0-11c5-4f7e-add8-707f5640cb28" containerID="f9a50618e15191809718222826c80cdd3b15d419f4bca07640f1324b5f5b9529" exitCode=0 Sep 10 19:11:02.748590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.748461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerDied","Data":"f9a50618e15191809718222826c80cdd3b15d419f4bca07640f1324b5f5b9529"} Sep 10 19:11:02.749125 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.749099 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:11:02.749218 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.749179 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:11:02.749585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.749229 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:11:02.764845 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.764823 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:11:02.764970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:02.764911 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:11:03.581636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:03.581399 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qxlb7"] Sep 10 19:11:03.581825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:03.581812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:03.581938 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:03.581911 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:11:03.583995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:03.583965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xrwb"] Sep 10 19:11:03.584099 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:03.584066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:03.584174 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:03.584153 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:11:03.585771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:03.585747 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sr4z7"] Sep 10 19:11:03.585866 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:03.585853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:03.585984 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:03.585962 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:11:04.754151 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:04.754113 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d3d0fc0-11c5-4f7e-add8-707f5640cb28" containerID="bef66e43ace4fde3622b5d5b870edf308022e7093b00b15ee8fd4859f6b71030" exitCode=0 Sep 10 19:11:04.754513 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:04.754196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerDied","Data":"bef66e43ace4fde3622b5d5b870edf308022e7093b00b15ee8fd4859f6b71030"} Sep 10 19:11:05.596248 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:05.596222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:05.596363 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:05.596331 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:11:05.596363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:05.596333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:05.596484 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:05.596409 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:11:05.596484 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:05.596337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:05.596484 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:05.596461 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:11:05.759585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:05.759499 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d3d0fc0-11c5-4f7e-add8-707f5640cb28" containerID="7d7d64348747fbf77b45cdb7433df0c6431021cb96facc3b6324e73a1f2b7b20" exitCode=0 Sep 10 19:11:05.759585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:05.759545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerDied","Data":"7d7d64348747fbf77b45cdb7433df0c6431021cb96facc3b6324e73a1f2b7b20"} Sep 10 19:11:07.596629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:07.596393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:07.597069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:07.596434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:07.597069 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:07.596744 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:11:07.597069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:07.596446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:07.597069 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:07.596815 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:11:07.597069 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:07.596893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:11:08.243644 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:08.243605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:08.243824 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.243784 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:11:08.243894 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.243864 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs podName:04d5421b-2ce5-469d-98e5-9fc2059de454 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:40.243842822 +0000 UTC m=+66.160498849 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs") pod "network-metrics-daemon-sr4z7" (UID: "04d5421b-2ce5-469d-98e5-9fc2059de454") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:11:08.344700 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:08.344604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:08.344700 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:08.344669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:08.344900 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.344779 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:11:08.344900 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.344796 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:11:08.344900 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.344820 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:11:08.344900 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.344830 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jzqqs for pod openshift-network-diagnostics/network-check-target-4xrwb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:11:08.344900 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.344857 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret podName:b14c0d55-d660-472c-ad93-f42bf3893fe9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:40.344837228 +0000 UTC m=+66.261493239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret") pod "global-pull-secret-syncer-qxlb7" (UID: "b14c0d55-d660-472c-ad93-f42bf3893fe9") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:11:08.344900 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:08.344875 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs podName:6a2c4140-61fe-483c-b71f-3d802b27a08f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:40.344863872 +0000 UTC m=+66.261519879 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzqqs" (UniqueName: "kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs") pod "network-check-target-4xrwb" (UID: "6a2c4140-61fe-483c-b71f-3d802b27a08f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:11:09.596546 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.596507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:09.597038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.596562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:09.597038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.596630 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:09.597038 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:09.596642 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qxlb7" podUID="b14c0d55-d660-472c-ad93-f42bf3893fe9" Sep 10 19:11:09.597038 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:09.596743 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr4z7" podUID="04d5421b-2ce5-469d-98e5-9fc2059de454" Sep 10 19:11:09.597038 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:09.596829 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xrwb" podUID="6a2c4140-61fe-483c-b71f-3d802b27a08f" Sep 10 19:11:09.870972 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.870946 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-43.ec2.internal" event="NodeReady" Sep 10 19:11:09.871141 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.871068 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 19:11:09.906856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.906828 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl"] Sep 10 19:11:09.947508 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.946727 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bdb4b6644-2b49n"] Sep 10 19:11:09.947508 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.946848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:09.950981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.950945 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-99h89\"" Sep 10 19:11:09.951138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.951018 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 19:11:09.953779 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.952116 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 19:11:09.953779 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.952291 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 19:11:09.954311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.954291 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 19:11:09.962852 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.962830 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-slhxt"] Sep 10 19:11:09.963523 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.963500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:09.967488 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.967467 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 19:11:09.967891 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.967874 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 19:11:09.968473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.968457 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8474t\"" Sep 10 19:11:09.968604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.968584 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 19:11:09.980868 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.980849 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-588979fc78-gdjfk"] Sep 10 19:11:09.980988 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.980935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:09.985886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.985847 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 19:11:09.986102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.986089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hxr4t\"" Sep 10 19:11:09.986900 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.986882 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 19:11:09.987107 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.987088 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 19:11:09.987748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.987733 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 19:11:09.995200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.995178 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 19:11:09.996113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.996092 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj"] Sep 10 19:11:09.996214 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.996154 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 19:11:09.996214 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.996194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:09.998932 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.998884 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 19:11:09.999028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.998971 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wh5bc\"" Sep 10 19:11:09.999213 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.999197 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 19:11:09.999433 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.999420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 19:11:09.999912 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.999893 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 19:11:09.999912 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.999908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 19:11:10.000025 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:09.999911 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.016236 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.016209 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr"] Sep 10 19:11:10.016395 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.016348 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:10.019708 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.019677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 19:11:10.019842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.019710 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:11:10.019842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.019777 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.019842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.019791 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gnzmg\"" Sep 10 19:11:10.038258 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.038232 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn"] Sep 10 19:11:10.038442 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.038346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" Sep 10 19:11:10.040875 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.040853 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:11:10.040989 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.040879 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.040989 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.040855 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k7d7r\"" Sep 10 19:11:10.056902 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.056871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec9ceda2-9fc5-439b-86c2-171a6effa80d-tmp\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.057050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.056912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b526d36-86c1-4af9-a501-7be9605f0e5d-ca-trust-extracted\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.057050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.056932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec9ceda2-9fc5-439b-86c2-171a6effa80d-snapshots\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.057050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.056950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-certificates\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.057050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec9ceda2-9fc5-439b-86c2-171a6effa80d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.057246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-image-registry-private-configuration\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.057246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec9ceda2-9fc5-439b-86c2-171a6effa80d-serving-cert\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.057246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9kt\" (UniqueName: \"kubernetes.io/projected/ec9ceda2-9fc5-439b-86c2-171a6effa80d-kube-api-access-8l9kt\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.057246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.057474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzqh\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-kube-api-access-rkzqh\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.057474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec9ceda2-9fc5-439b-86c2-171a6effa80d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.057474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-trusted-ca\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.057474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.057474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-installation-pull-secrets\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.057474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.057725 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5cn\" (UniqueName: \"kubernetes.io/projected/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-kube-api-access-7g5cn\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.057725 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.057519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-bound-sa-token\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.064403 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.064358 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-lq654"] Sep 10 19:11:10.064537 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.064479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.067019 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.066999 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 19:11:10.067134 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.067004 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2hzwf\"" Sep 10 19:11:10.067200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.067155 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.067314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.067296 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:11:10.067406 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.067340 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 19:11:10.076532 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.076505 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b"] Sep 10 19:11:10.076642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.076628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.079360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.079300 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.079360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.079344 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 19:11:10.079617 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.079344 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hxvtg\"" Sep 10 19:11:10.079717 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.079652 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:11:10.080053 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.080036 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 19:11:10.092755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl"] Sep 10 19:11:10.092847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092765 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-slhxt"] Sep 10 19:11:10.092847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092796 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr"] Sep 10 19:11:10.092847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn"] Sep 10 19:11:10.092847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bdb4b6644-2b49n"] Sep 10 19:11:10.092847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092834 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj"] Sep 10 19:11:10.092847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.093113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.092849 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-588979fc78-gdjfk"] Sep 10 19:11:10.093113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.093003 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-twfjs"] Sep 10 19:11:10.096163 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.095559 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:11:10.096163 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.095632 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 19:11:10.096163 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.095795 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 19:11:10.096163 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.095812 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pflwn\"" Sep 10 19:11:10.096163 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.096008 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.096755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.096732 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 19:11:10.112890 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.112866 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b"] Sep 10 19:11:10.113032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.112896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" Sep 10 19:11:10.114315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.113128 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-lq654"] Sep 10 19:11:10.114315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.113879 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v4j2x"] Sep 10 19:11:10.115950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.115929 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 19:11:10.116064 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.115989 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.116064 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.115941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6knsk\"" Sep 10 19:11:10.133455 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.133434 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t7gm6"] Sep 10 19:11:10.133567 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.133553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.135954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.135933 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zn8jq\"" Sep 10 19:11:10.136072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.135954 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 19:11:10.136072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.135967 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 19:11:10.136072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.135967 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 19:11:10.143501 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.143481 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-twfjs"] Sep 10 19:11:10.143609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.143510 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v4j2x"] Sep 10 19:11:10.143609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.143530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t7gm6"] Sep 10 19:11:10.143609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.143593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.145960 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.145941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 19:11:10.146072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.145996 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 19:11:10.146137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.145947 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b59cd\"" Sep 10 19:11:10.158346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec9ceda2-9fc5-439b-86c2-171a6effa80d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.158479 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-stats-auth\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.158479 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec9ceda2-9fc5-439b-86c2-171a6effa80d-serving-cert\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.158584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.158584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-installation-pull-secrets\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.158584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9kt\" (UniqueName: \"kubernetes.io/projected/ec9ceda2-9fc5-439b-86c2-171a6effa80d-kube-api-access-8l9kt\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.158584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzqh\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-kube-api-access-rkzqh\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.158759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-serving-cert\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.158759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/226486d2-fe60-4143-9151-43c5cae901ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.158759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec9ceda2-9fc5-439b-86c2-171a6effa80d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.158759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.158759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-default-certificate\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.158981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5cn\" (UniqueName: \"kubernetes.io/projected/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-kube-api-access-7g5cn\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.158981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.158981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec9ceda2-9fc5-439b-86c2-171a6effa80d-tmp\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.158981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j898w\" (UniqueName: \"kubernetes.io/projected/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-kube-api-access-j898w\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.158981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc51729-75fb-49a9-a31e-827cde63ffc1-config\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.158981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b526d36-86c1-4af9-a501-7be9605f0e5d-ca-trust-extracted\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.158981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.158974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhc8\" (UniqueName: \"kubernetes.io/projected/06758258-e762-4934-9054-deda9c74df12-kube-api-access-vfhc8\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:10.159714 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.159688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec9ceda2-9fc5-439b-86c2-171a6effa80d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.159808 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.159737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec9ceda2-9fc5-439b-86c2-171a6effa80d-snapshots\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.159983 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.159964 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 19:11:10.160088 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.159984 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdb4b6644-2b49n: secret "image-registry-tls" not found Sep 10 19:11:10.160088 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.160056 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls podName:6b526d36-86c1-4af9-a501-7be9605f0e5d nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.660036783 +0000 UTC m=+36.576692790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls") pod "image-registry-7bdb4b6644-2b49n" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d") : secret "image-registry-tls" not found Sep 10 19:11:10.160855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.160822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-certificates\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.161549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec9ceda2-9fc5-439b-86c2-171a6effa80d-tmp\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.161549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec9ceda2-9fc5-439b-86c2-171a6effa80d-snapshots\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.161549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b526d36-86c1-4af9-a501-7be9605f0e5d-ca-trust-extracted\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.161549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-image-registry-private-configuration\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.161549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-certificates\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.161549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc51729-75fb-49a9-a31e-827cde63ffc1-serving-cert\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.161572 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/226486d2-fe60-4143-9151-43c5cae901ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-config\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.161669 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls podName:8d4d0730-72f3-4a8f-b169-b008b59ad4b0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.661644022 +0000 UTC m=+36.578300043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pnvfl" (UID: "8d4d0730-72f3-4a8f-b169-b008b59ad4b0") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec9ceda2-9fc5-439b-86c2-171a6effa80d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-trusted-ca\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-trusted-ca\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.161898 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.162261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-bound-sa-token\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.162261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62hs\" (UniqueName: \"kubernetes.io/projected/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-kube-api-access-p62hs\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.162261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.161985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx56s\" (UniqueName: \"kubernetes.io/projected/226486d2-fe60-4143-9151-43c5cae901ee-kube-api-access-fx56s\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.162261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.162012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5js6\" (UniqueName: \"kubernetes.io/projected/e5516b97-0f82-4a19-86a6-ce51e793fddb-kube-api-access-x5js6\") pod \"volume-data-source-validator-695c895bd6-zxqxr\" (UID: \"e5516b97-0f82-4a19-86a6-ce51e793fddb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" Sep 10 19:11:10.162261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.162044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:10.162261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.162078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqw7s\" (UniqueName: \"kubernetes.io/projected/cdc51729-75fb-49a9-a31e-827cde63ffc1-kube-api-access-wqw7s\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.162806 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.162784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-trusted-ca\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.163448 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.163398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.164908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.164885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec9ceda2-9fc5-439b-86c2-171a6effa80d-serving-cert\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.164995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.164905 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-installation-pull-secrets\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.165780 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.165759 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-image-registry-private-configuration\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.167244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.167224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9kt\" (UniqueName: \"kubernetes.io/projected/ec9ceda2-9fc5-439b-86c2-171a6effa80d-kube-api-access-8l9kt\") pod \"insights-operator-76bcd6ccf5-slhxt\" (UID: \"ec9ceda2-9fc5-439b-86c2-171a6effa80d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.170256 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.170208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5cn\" (UniqueName: \"kubernetes.io/projected/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-kube-api-access-7g5cn\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.170358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.170295 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzqh\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-kube-api-access-rkzqh\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.170358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.170385 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-bound-sa-token\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.263078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.262997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-serving-cert\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.263078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/226486d2-fe60-4143-9151-43c5cae901ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.263078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-default-certificate\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j898w\" (UniqueName: \"kubernetes.io/projected/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-kube-api-access-j898w\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc51729-75fb-49a9-a31e-827cde63ffc1-config\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhc8\" (UniqueName: \"kubernetes.io/projected/06758258-e762-4934-9054-deda9c74df12-kube-api-access-vfhc8\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.263214 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.76319017 +0000 UTC m=+36.679846176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : configmap references non-existent config key: service-ca.crt Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.263258 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ff1c36-92f9-4f39-82da-95672fcd5de9-tmp-dir\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.263341 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.263303 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.763291294 +0000 UTC m=+36.679947301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : secret "router-metrics-certs-default" not found Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ncp\" (UniqueName: \"kubernetes.io/projected/124cc65e-d0ed-4659-99d9-cb8a2c7a8763-kube-api-access-f2ncp\") pod \"ingress-canary-v4j2x\" (UID: \"124cc65e-d0ed-4659-99d9-cb8a2c7a8763\") " pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zthd\" (UniqueName: \"kubernetes.io/projected/d9ff1c36-92f9-4f39-82da-95672fcd5de9-kube-api-access-5zthd\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/124cc65e-d0ed-4659-99d9-cb8a2c7a8763-cert\") pod \"ingress-canary-v4j2x\" (UID: \"124cc65e-d0ed-4659-99d9-cb8a2c7a8763\") " pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc51729-75fb-49a9-a31e-827cde63ffc1-serving-cert\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/226486d2-fe60-4143-9151-43c5cae901ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-config\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-trusted-ca\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ff1c36-92f9-4f39-82da-95672fcd5de9-config-volume\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/226486d2-fe60-4143-9151-43c5cae901ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvk2\" (UniqueName: \"kubernetes.io/projected/f8128405-cc96-49e2-9d8f-eb3a3088affd-kube-api-access-vsvk2\") pod \"network-check-source-685456655f-twfjs\" (UID: \"f8128405-cc96-49e2-9d8f-eb3a3088affd\") " pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62hs\" (UniqueName: \"kubernetes.io/projected/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-kube-api-access-p62hs\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx56s\" (UniqueName: \"kubernetes.io/projected/226486d2-fe60-4143-9151-43c5cae901ee-kube-api-access-fx56s\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5js6\" (UniqueName: \"kubernetes.io/projected/e5516b97-0f82-4a19-86a6-ce51e793fddb-kube-api-access-x5js6\") pod \"volume-data-source-validator-695c895bd6-zxqxr\" (UID: \"e5516b97-0f82-4a19-86a6-ce51e793fddb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" Sep 10 19:11:10.263818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc51729-75fb-49a9-a31e-827cde63ffc1-config\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.264790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:10.264790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqw7s\" (UniqueName: \"kubernetes.io/projected/cdc51729-75fb-49a9-a31e-827cde63ffc1-kube-api-access-wqw7s\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.264790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.263898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-stats-auth\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.264790 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.264443 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 19:11:10.264790 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.264506 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls podName:06758258-e762-4934-9054-deda9c74df12 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.76448945 +0000 UTC m=+36.681145458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-26rsj" (UID: "06758258-e762-4934-9054-deda9c74df12") : secret "samples-operator-tls" not found Sep 10 19:11:10.264790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.264753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-trusted-ca\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.265202 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.265179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-config\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.266251 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.266229 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc51729-75fb-49a9-a31e-827cde63ffc1-serving-cert\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.266344 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.266273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-default-certificate\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.266344 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.266338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-serving-cert\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.266480 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.266465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/226486d2-fe60-4143-9151-43c5cae901ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.266526 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.266496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-stats-auth\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.274020 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.273991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhc8\" (UniqueName: \"kubernetes.io/projected/06758258-e762-4934-9054-deda9c74df12-kube-api-access-vfhc8\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:10.275199 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.275174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx56s\" (UniqueName: \"kubernetes.io/projected/226486d2-fe60-4143-9151-43c5cae901ee-kube-api-access-fx56s\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwbxn\" (UID: \"226486d2-fe60-4143-9151-43c5cae901ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.275473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.275448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqw7s\" (UniqueName: \"kubernetes.io/projected/cdc51729-75fb-49a9-a31e-827cde63ffc1-kube-api-access-wqw7s\") pod \"service-ca-operator-859f94c4b-zrw7b\" (UID: \"cdc51729-75fb-49a9-a31e-827cde63ffc1\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.275566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.275483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j898w\" (UniqueName: \"kubernetes.io/projected/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-kube-api-access-j898w\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.275806 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.275788 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5js6\" (UniqueName: \"kubernetes.io/projected/e5516b97-0f82-4a19-86a6-ce51e793fddb-kube-api-access-x5js6\") pod \"volume-data-source-validator-695c895bd6-zxqxr\" (UID: \"e5516b97-0f82-4a19-86a6-ce51e793fddb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" Sep 10 19:11:10.276524 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.276498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62hs\" (UniqueName: \"kubernetes.io/projected/264d6101-b27d-4c83-bfe3-f70d9dcb0f74-kube-api-access-p62hs\") pod \"console-operator-85b454874b-lq654\" (UID: \"264d6101-b27d-4c83-bfe3-f70d9dcb0f74\") " pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.291461 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.291438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" Sep 10 19:11:10.349495 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.349463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" Sep 10 19:11:10.364991 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.364957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ff1c36-92f9-4f39-82da-95672fcd5de9-config-volume\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.365070 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.365070 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvk2\" (UniqueName: \"kubernetes.io/projected/f8128405-cc96-49e2-9d8f-eb3a3088affd-kube-api-access-vsvk2\") pod \"network-check-source-685456655f-twfjs\" (UID: \"f8128405-cc96-49e2-9d8f-eb3a3088affd\") " pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" Sep 10 19:11:10.365169 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.365147 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:11:10.365222 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ff1c36-92f9-4f39-82da-95672fcd5de9-tmp-dir\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.365222 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ncp\" (UniqueName: \"kubernetes.io/projected/124cc65e-d0ed-4659-99d9-cb8a2c7a8763-kube-api-access-f2ncp\") pod \"ingress-canary-v4j2x\" (UID: \"124cc65e-d0ed-4659-99d9-cb8a2c7a8763\") " pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.365324 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.365217 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls podName:d9ff1c36-92f9-4f39-82da-95672fcd5de9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.865195528 +0000 UTC m=+36.781851557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls") pod "dns-default-t7gm6" (UID: "d9ff1c36-92f9-4f39-82da-95672fcd5de9") : secret "dns-default-metrics-tls" not found Sep 10 19:11:10.365324 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zthd\" (UniqueName: \"kubernetes.io/projected/d9ff1c36-92f9-4f39-82da-95672fcd5de9-kube-api-access-5zthd\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.365324 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/124cc65e-d0ed-4659-99d9-cb8a2c7a8763-cert\") pod \"ingress-canary-v4j2x\" (UID: \"124cc65e-d0ed-4659-99d9-cb8a2c7a8763\") " pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.365494 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365481 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ff1c36-92f9-4f39-82da-95672fcd5de9-tmp-dir\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.365698 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.365663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ff1c36-92f9-4f39-82da-95672fcd5de9-config-volume\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.368970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.368902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/124cc65e-d0ed-4659-99d9-cb8a2c7a8763-cert\") pod \"ingress-canary-v4j2x\" (UID: \"124cc65e-d0ed-4659-99d9-cb8a2c7a8763\") " pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.373342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.373315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" Sep 10 19:11:10.376014 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.375984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zthd\" (UniqueName: \"kubernetes.io/projected/d9ff1c36-92f9-4f39-82da-95672fcd5de9-kube-api-access-5zthd\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.376232 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.376216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvk2\" (UniqueName: \"kubernetes.io/projected/f8128405-cc96-49e2-9d8f-eb3a3088affd-kube-api-access-vsvk2\") pod \"network-check-source-685456655f-twfjs\" (UID: \"f8128405-cc96-49e2-9d8f-eb3a3088affd\") " pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" Sep 10 19:11:10.376366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.376349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ncp\" (UniqueName: \"kubernetes.io/projected/124cc65e-d0ed-4659-99d9-cb8a2c7a8763-kube-api-access-f2ncp\") pod \"ingress-canary-v4j2x\" (UID: \"124cc65e-d0ed-4659-99d9-cb8a2c7a8763\") " pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.387227 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.387200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:10.403573 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.403489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" Sep 10 19:11:10.424277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.424241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" Sep 10 19:11:10.443243 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.443206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v4j2x" Sep 10 19:11:10.626238 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.626207 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-slhxt"] Sep 10 19:11:10.635583 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.635427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr"] Sep 10 19:11:10.667568 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.667541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:10.667712 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.667633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:10.667712 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.667691 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:10.667823 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.667747 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 19:11:10.667823 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.667757 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdb4b6644-2b49n: secret "image-registry-tls" not found Sep 10 19:11:10.667823 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.667760 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls podName:8d4d0730-72f3-4a8f-b169-b008b59ad4b0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.667742236 +0000 UTC m=+37.584398244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pnvfl" (UID: "8d4d0730-72f3-4a8f-b169-b008b59ad4b0") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:10.667823 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.667798 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls podName:6b526d36-86c1-4af9-a501-7be9605f0e5d nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.667786587 +0000 UTC m=+37.584442608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls") pod "image-registry-7bdb4b6644-2b49n" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d") : secret "image-registry-tls" not found Sep 10 19:11:10.684642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.684592 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn"] Sep 10 19:11:10.706541 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:10.706508 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9ceda2_9fc5_439b_86c2_171a6effa80d.slice/crio-ec02033e8508c5ce201642bc9362981435ffa235def81c69ac2bdc1fcfd231fc WatchSource:0}: Error finding container ec02033e8508c5ce201642bc9362981435ffa235def81c69ac2bdc1fcfd231fc: Status 404 returned error can't find the container with id ec02033e8508c5ce201642bc9362981435ffa235def81c69ac2bdc1fcfd231fc Sep 10 19:11:10.707238 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:10.707105 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5516b97_0f82_4a19_86a6_ce51e793fddb.slice/crio-a8db61a216d4421bbf574a056a6a8d2d2895e0f250b281520d3c725bfc088415 WatchSource:0}: Error finding container a8db61a216d4421bbf574a056a6a8d2d2895e0f250b281520d3c725bfc088415: Status 404 returned error can't find the container with id a8db61a216d4421bbf574a056a6a8d2d2895e0f250b281520d3c725bfc088415 Sep 10 19:11:10.707839 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:10.707818 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226486d2_fe60_4143_9151_43c5cae901ee.slice/crio-2637ccb05022b7ded2b053031772ee8254789019f27ab0ab5bf5965551e76532 WatchSource:0}: Error finding container 2637ccb05022b7ded2b053031772ee8254789019f27ab0ab5bf5965551e76532: Status 404 returned error can't find the container with id 2637ccb05022b7ded2b053031772ee8254789019f27ab0ab5bf5965551e76532 Sep 10 19:11:10.716565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.716546 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-lq654"] Sep 10 19:11:10.718445 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:10.718427 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264d6101_b27d_4c83_bfe3_f70d9dcb0f74.slice/crio-b9f6fe674accbf44dbab9963a4a16845e294036fff57a8c290bbf60b40ea8636 WatchSource:0}: Error finding container b9f6fe674accbf44dbab9963a4a16845e294036fff57a8c290bbf60b40ea8636: Status 404 returned error can't find the container with id b9f6fe674accbf44dbab9963a4a16845e294036fff57a8c290bbf60b40ea8636 Sep 10 19:11:10.746182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.746160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-twfjs"] Sep 10 19:11:10.748667 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:10.748647 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8128405_cc96_49e2_9d8f_eb3a3088affd.slice/crio-aac1a614bfb41f242637965da9b5518bb3b86edc92332e4939f3b03a63a49186 WatchSource:0}: Error finding container aac1a614bfb41f242637965da9b5518bb3b86edc92332e4939f3b03a63a49186: Status 404 returned error can't find the container with id aac1a614bfb41f242637965da9b5518bb3b86edc92332e4939f3b03a63a49186 Sep 10 19:11:10.768818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.768773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.768940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.768851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:10.769000 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.768942 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.768923617 +0000 UTC m=+37.685579644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : configmap references non-existent config key: service-ca.crt Sep 10 19:11:10.769000 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.768993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:10.769129 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.769011 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:11:10.769129 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.769068 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.769049307 +0000 UTC m=+37.685705334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : secret "router-metrics-certs-default" not found Sep 10 19:11:10.769129 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.769104 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 19:11:10.769291 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.769143 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls podName:06758258-e762-4934-9054-deda9c74df12 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.769131776 +0000 UTC m=+37.685787783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-26rsj" (UID: "06758258-e762-4934-9054-deda9c74df12") : secret "samples-operator-tls" not found Sep 10 19:11:10.769998 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.769973 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v4j2x"] Sep 10 19:11:10.773382 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:10.773335 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124cc65e_d0ed_4659_99d9_cb8a2c7a8763.slice/crio-335bbd279415b69d4a656f1d1162fa226f73a6410a1443c221802b67b9b3e91d WatchSource:0}: Error finding container 335bbd279415b69d4a656f1d1162fa226f73a6410a1443c221802b67b9b3e91d: Status 404 returned error can't find the container with id 335bbd279415b69d4a656f1d1162fa226f73a6410a1443c221802b67b9b3e91d Sep 10 19:11:10.773651 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.773626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lq654" event={"ID":"264d6101-b27d-4c83-bfe3-f70d9dcb0f74","Type":"ContainerStarted","Data":"b9f6fe674accbf44dbab9963a4a16845e294036fff57a8c290bbf60b40ea8636"} Sep 10 19:11:10.773780 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.773760 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b"] Sep 10 19:11:10.774903 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.774884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" event={"ID":"226486d2-fe60-4143-9151-43c5cae901ee","Type":"ContainerStarted","Data":"2637ccb05022b7ded2b053031772ee8254789019f27ab0ab5bf5965551e76532"} Sep 10 19:11:10.776496 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.776475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" event={"ID":"e5516b97-0f82-4a19-86a6-ce51e793fddb","Type":"ContainerStarted","Data":"a8db61a216d4421bbf574a056a6a8d2d2895e0f250b281520d3c725bfc088415"} Sep 10 19:11:10.777174 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:10.777153 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc51729_75fb_49a9_a31e_827cde63ffc1.slice/crio-ba84be33f8c3a70589df0c7eae84c573b9381caa5ec3e13d3945f33cc5886a00 WatchSource:0}: Error finding container ba84be33f8c3a70589df0c7eae84c573b9381caa5ec3e13d3945f33cc5886a00: Status 404 returned error can't find the container with id ba84be33f8c3a70589df0c7eae84c573b9381caa5ec3e13d3945f33cc5886a00 Sep 10 19:11:10.777805 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.777781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" event={"ID":"ec9ceda2-9fc5-439b-86c2-171a6effa80d","Type":"ContainerStarted","Data":"ec02033e8508c5ce201642bc9362981435ffa235def81c69ac2bdc1fcfd231fc"} Sep 10 19:11:10.778998 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.778976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" event={"ID":"f8128405-cc96-49e2-9d8f-eb3a3088affd","Type":"ContainerStarted","Data":"aac1a614bfb41f242637965da9b5518bb3b86edc92332e4939f3b03a63a49186"} Sep 10 19:11:10.869983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:10.869960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:10.870153 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.870132 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:11:10.870221 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:10.870212 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls podName:d9ff1c36-92f9-4f39-82da-95672fcd5de9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.870190307 +0000 UTC m=+37.786846331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls") pod "dns-default-t7gm6" (UID: "d9ff1c36-92f9-4f39-82da-95672fcd5de9") : secret "dns-default-metrics-tls" not found Sep 10 19:11:11.598198 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.596661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:11.598198 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.597156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:11.598198 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.597627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:11.602314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.602276 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 19:11:11.602557 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.602537 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zj5gq\"" Sep 10 19:11:11.602740 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.602724 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 19:11:11.602933 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.602915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gdr\"" Sep 10 19:11:11.677242 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.677201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:11.677750 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.677333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:11.677750 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.677535 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 19:11:11.677750 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.677552 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdb4b6644-2b49n: secret "image-registry-tls" not found Sep 10 19:11:11.677750 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.677615 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls podName:6b526d36-86c1-4af9-a501-7be9605f0e5d nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.677591488 +0000 UTC m=+39.594247511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls") pod "image-registry-7bdb4b6644-2b49n" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d") : secret "image-registry-tls" not found Sep 10 19:11:11.678079 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.678061 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:11.678133 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.678116 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls podName:8d4d0730-72f3-4a8f-b169-b008b59ad4b0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.678102136 +0000 UTC m=+39.594758149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pnvfl" (UID: "8d4d0730-72f3-4a8f-b169-b008b59ad4b0") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:11.778235 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.778200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:11.778436 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.778255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:11.778436 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.778403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:11.778570 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.778554 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 19:11:11.778633 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.778622 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls podName:06758258-e762-4934-9054-deda9c74df12 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.778602738 +0000 UTC m=+39.695258753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-26rsj" (UID: "06758258-e762-4934-9054-deda9c74df12") : secret "samples-operator-tls" not found Sep 10 19:11:11.779065 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.779044 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.779026998 +0000 UTC m=+39.695683011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : configmap references non-existent config key: service-ca.crt Sep 10 19:11:11.779138 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.779123 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:11:11.779186 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.779158 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.779146682 +0000 UTC m=+39.695802697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : secret "router-metrics-certs-default" not found Sep 10 19:11:11.788227 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.788194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v4j2x" event={"ID":"124cc65e-d0ed-4659-99d9-cb8a2c7a8763","Type":"ContainerStarted","Data":"335bbd279415b69d4a656f1d1162fa226f73a6410a1443c221802b67b9b3e91d"} Sep 10 19:11:11.793988 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.793960 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d3d0fc0-11c5-4f7e-add8-707f5640cb28" containerID="f325505dc3dc6ccedb33b94e16230b6732baa52a060c7bf4af4ee32c2845f32e" exitCode=0 Sep 10 19:11:11.794125 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.794067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerDied","Data":"f325505dc3dc6ccedb33b94e16230b6732baa52a060c7bf4af4ee32c2845f32e"} Sep 10 19:11:11.797241 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.797165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" event={"ID":"cdc51729-75fb-49a9-a31e-827cde63ffc1","Type":"ContainerStarted","Data":"ba84be33f8c3a70589df0c7eae84c573b9381caa5ec3e13d3945f33cc5886a00"} Sep 10 19:11:11.879936 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:11.879849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:11.880262 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.880239 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:11:11.880366 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:11.880312 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls podName:d9ff1c36-92f9-4f39-82da-95672fcd5de9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.880291664 +0000 UTC m=+39.796947671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls") pod "dns-default-t7gm6" (UID: "d9ff1c36-92f9-4f39-82da-95672fcd5de9") : secret "dns-default-metrics-tls" not found Sep 10 19:11:12.811260 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:12.811220 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d3d0fc0-11c5-4f7e-add8-707f5640cb28" containerID="bf5bc9337ea45b1cb8e7d944befdfa3568116fc02426c092a4b3f005531bcdd4" exitCode=0 Sep 10 19:11:12.811805 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:12.811287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerDied","Data":"bf5bc9337ea45b1cb8e7d944befdfa3568116fc02426c092a4b3f005531bcdd4"} Sep 10 19:11:13.652728 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.652688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:13.699885 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.699846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:13.700071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.699975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:13.700071 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.700007 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 19:11:13.700071 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.700022 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdb4b6644-2b49n: secret "image-registry-tls" not found Sep 10 19:11:13.700239 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.700083 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls podName:6b526d36-86c1-4af9-a501-7be9605f0e5d nodeName:}" failed. No retries permitted until 2026-09-10 19:11:17.700066848 +0000 UTC m=+43.616722857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls") pod "image-registry-7bdb4b6644-2b49n" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d") : secret "image-registry-tls" not found Sep 10 19:11:13.700239 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.700173 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:13.700239 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.700209 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls podName:8d4d0730-72f3-4a8f-b169-b008b59ad4b0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:17.700198653 +0000 UTC m=+43.616854661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pnvfl" (UID: "8d4d0730-72f3-4a8f-b169-b008b59ad4b0") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:13.737068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.736977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:13.801270 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.801230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:13.801467 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.801364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:13.801467 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.801463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:13.801627 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.801613 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:17.801592991 +0000 UTC m=+43.718249014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : configmap references non-existent config key: service-ca.crt Sep 10 19:11:13.801720 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.801706 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:11:13.801779 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.801746 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:17.801734714 +0000 UTC m=+43.718390738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : secret "router-metrics-certs-default" not found Sep 10 19:11:13.801832 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.801803 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 19:11:13.801868 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.801833 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls podName:06758258-e762-4934-9054-deda9c74df12 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:17.801822875 +0000 UTC m=+43.718478929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-26rsj" (UID: "06758258-e762-4934-9054-deda9c74df12") : secret "samples-operator-tls" not found Sep 10 19:11:13.819285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.819254 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" event={"ID":"7d3d0fc0-11c5-4f7e-add8-707f5640cb28","Type":"ContainerStarted","Data":"fd11c40871cb7d2188e5dcf54c739d06412daee40be813a9c076e23f343943eb"} Sep 10 19:11:13.840925 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.840874 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dnd8d" podStartSLOduration=9.501289524 podStartE2EDuration="39.840856698s" totalImagesPullingTime="30.339567174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:13.839401853 +0000 UTC m=+39.756057912" watchObservedRunningTime="2026-09-10 19:11:13.840856698 +0000 UTC m=+39.757512727" Sep 10 19:11:13.902505 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.902469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:13.902685 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.902646 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:11:13.902743 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:13.902717 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls podName:d9ff1c36-92f9-4f39-82da-95672fcd5de9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:17.902697588 +0000 UTC m=+43.819353617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls") pod "dns-default-t7gm6" (UID: "d9ff1c36-92f9-4f39-82da-95672fcd5de9") : secret "dns-default-metrics-tls" not found Sep 10 19:11:13.962139 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:13.962107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:14.107577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:14.107549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:14.165523 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:14.165479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:14.856797 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:14.856467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:14.913613 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:14.913574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:16.288732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:16.288703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:16.343874 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:16.343836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:17.056886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.056853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:17.113809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.113774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:17.215453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.215412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:17.354830 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.354693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:17.734733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.734657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:17.734879 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.734754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:17.734879 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.734804 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 19:11:17.734879 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.734822 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdb4b6644-2b49n: secret "image-registry-tls" not found Sep 10 19:11:17.734879 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.734867 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:17.734879 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.734878 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls podName:6b526d36-86c1-4af9-a501-7be9605f0e5d nodeName:}" failed. No retries permitted until 2026-09-10 19:11:25.734860883 +0000 UTC m=+51.651516904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls") pod "image-registry-7bdb4b6644-2b49n" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d") : secret "image-registry-tls" not found Sep 10 19:11:17.735051 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.734906 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls podName:8d4d0730-72f3-4a8f-b169-b008b59ad4b0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:25.734894691 +0000 UTC m=+51.651550697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pnvfl" (UID: "8d4d0730-72f3-4a8f-b169-b008b59ad4b0") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:17.835722 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.835689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:17.835897 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.835755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:17.835897 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.835781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:17.835897 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.835830 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 19:11:17.835897 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.835884 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls podName:06758258-e762-4934-9054-deda9c74df12 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:25.835868986 +0000 UTC m=+51.752525016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-26rsj" (UID: "06758258-e762-4934-9054-deda9c74df12") : secret "samples-operator-tls" not found Sep 10 19:11:17.836068 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.835909 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:11:17.836068 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.835909 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:25.835890957 +0000 UTC m=+51.752546982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : configmap references non-existent config key: service-ca.crt Sep 10 19:11:17.836068 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.835941 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:25.835930953 +0000 UTC m=+51.752586960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : secret "router-metrics-certs-default" not found Sep 10 19:11:17.937192 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:17.937148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:17.937339 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.937303 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:11:17.937436 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:17.937365 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls podName:d9ff1c36-92f9-4f39-82da-95672fcd5de9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:25.937351017 +0000 UTC m=+51.854007028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls") pod "dns-default-t7gm6" (UID: "d9ff1c36-92f9-4f39-82da-95672fcd5de9") : secret "dns-default-metrics-tls" not found Sep 10 19:11:19.498935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.498846 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:19.555478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.555445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:19.835364 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.835321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v4j2x" event={"ID":"124cc65e-d0ed-4659-99d9-cb8a2c7a8763","Type":"ContainerStarted","Data":"49ea0ed3c5591bb5cb6e374f9c1e5eb428e35d190d0ddeb4b6755186c3c056a6"} Sep 10 19:11:19.837503 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.837479 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/0.log" Sep 10 19:11:19.837642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.837517 2576 generic.go:356] "Generic (PLEG): container finished" podID="264d6101-b27d-4c83-bfe3-f70d9dcb0f74" containerID="7cc0578ad0253c31fc53dd75ef03891bc4f1fd806d1144d9efa840eedc7b4d6e" exitCode=255 Sep 10 19:11:19.837756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.837730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lq654" event={"ID":"264d6101-b27d-4c83-bfe3-f70d9dcb0f74","Type":"ContainerDied","Data":"7cc0578ad0253c31fc53dd75ef03891bc4f1fd806d1144d9efa840eedc7b4d6e"} Sep 10 19:11:19.837862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.837847 2576 scope.go:117] "RemoveContainer" containerID="7cc0578ad0253c31fc53dd75ef03891bc4f1fd806d1144d9efa840eedc7b4d6e" Sep 10 19:11:19.840106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.840042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" event={"ID":"226486d2-fe60-4143-9151-43c5cae901ee","Type":"ContainerStarted","Data":"d7050974e0cae1d9e845808b723ce80235d521ef6458cfcbd9f3da363113ee85"} Sep 10 19:11:19.845352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.845329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" event={"ID":"e5516b97-0f82-4a19-86a6-ce51e793fddb","Type":"ContainerStarted","Data":"974c1253a2f95cc3baac50ef2b0cddb5ab83360bdb010dee0eaae81f866400e2"} Sep 10 19:11:19.847237 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.847211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" event={"ID":"ec9ceda2-9fc5-439b-86c2-171a6effa80d","Type":"ContainerStarted","Data":"5a922f20286c7caa9d2a743cdc033e2138d96db8d2f58e13030a4671e0be3786"} Sep 10 19:11:19.848824 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.848797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" event={"ID":"cdc51729-75fb-49a9-a31e-827cde63ffc1","Type":"ContainerStarted","Data":"4b8f96afcee4d8923aee51d91750f6bea796e35145349139443ff29e66220a86"} Sep 10 19:11:19.850335 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.850311 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" event={"ID":"f8128405-cc96-49e2-9d8f-eb3a3088affd","Type":"ContainerStarted","Data":"f1c65ceb167b4e66fd7ff704ca928b862858e2e1bd95c455b86435f44af87226"} Sep 10 19:11:19.854022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.853973 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v4j2x" podStartSLOduration=2.807600057 podStartE2EDuration="10.853955144s" totalImagesPullingTime="8.046355087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:19.852279684 +0000 UTC m=+45.768935714" watchObservedRunningTime="2026-09-10 19:11:19.853955144 +0000 UTC m=+45.770611174" Sep 10 19:11:19.872809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.872512 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" podStartSLOduration=30.774696125 podStartE2EDuration="38.872494575s" totalImagesPullingTime="8.09779845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:19.871846745 +0000 UTC m=+45.788502776" watchObservedRunningTime="2026-09-10 19:11:19.872494575 +0000 UTC m=+45.789150612" Sep 10 19:11:19.891881 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.891832 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zxqxr" podStartSLOduration=29.133897267 podStartE2EDuration="36.891813844s" totalImagesPullingTime="7.757916577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:19.891543314 +0000 UTC m=+45.808199358" watchObservedRunningTime="2026-09-10 19:11:19.891813844 +0000 UTC m=+45.808469874" Sep 10 19:11:19.908439 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.907857 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" podStartSLOduration=23.809558308 podStartE2EDuration="31.907842449s" totalImagesPullingTime="8.098284141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:19.907545694 +0000 UTC m=+45.824201724" watchObservedRunningTime="2026-09-10 19:11:19.907842449 +0000 UTC m=+45.824498554" Sep 10 19:11:19.947616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.947565 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-twfjs" podStartSLOduration=21.876452284 podStartE2EDuration="29.947524841s" totalImagesPullingTime="8.071072557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:19.945993537 +0000 UTC m=+45.862649559" watchObservedRunningTime="2026-09-10 19:11:19.947524841 +0000 UTC m=+45.864180881" Sep 10 19:11:19.962469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.962431 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" podStartSLOduration=20.915027818 podStartE2EDuration="28.962415096s" totalImagesPullingTime="8.047387278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:19.962070163 +0000 UTC m=+45.878726185" watchObservedRunningTime="2026-09-10 19:11:19.962415096 +0000 UTC m=+45.879071124" Sep 10 19:11:19.976528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:19.976482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:20.147072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.146990 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-whqpq"] Sep 10 19:11:20.154052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.151162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:20.184707 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.184668 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-whqpq"] Sep 10 19:11:20.184863 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.184810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.187451 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.187428 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 19:11:20.187613 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.187596 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 19:11:20.187613 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.187607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zl4hx\"" Sep 10 19:11:20.219505 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.219478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:20.358915 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.358873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/996a8504-cb25-471d-a86e-f34bd0f9915f-crio-socket\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.359112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.358956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqpb\" (UniqueName: \"kubernetes.io/projected/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-api-access-ndqpb\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.359112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.358982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a8504-cb25-471d-a86e-f34bd0f9915f-data-volume\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.359112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.359063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.359112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.359099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.387890 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.387851 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:20.388031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.387922 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:20.460250 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.460164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.460250 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.460221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.460477 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.460295 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:11:20.460477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.460323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/996a8504-cb25-471d-a86e-f34bd0f9915f-crio-socket\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.460477 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.460392 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:20.96035345 +0000 UTC m=+46.877009481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : configmap "kube-rbac-proxy" not found Sep 10 19:11:20.460477 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.460401 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:11:20.460477 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.460441 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:20.960430695 +0000 UTC m=+46.877086717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : secret "insights-runtime-extractor-tls" not found Sep 10 19:11:20.460477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.460458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqpb\" (UniqueName: \"kubernetes.io/projected/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-api-access-ndqpb\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.460477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.460480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a8504-cb25-471d-a86e-f34bd0f9915f-data-volume\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.460747 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.460554 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/996a8504-cb25-471d-a86e-f34bd0f9915f-crio-socket\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.460822 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.460806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a8504-cb25-471d-a86e-f34bd0f9915f-data-volume\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.469757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.469737 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqpb\" (UniqueName: \"kubernetes.io/projected/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-api-access-ndqpb\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.794528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.794492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:20.855440 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.855408 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/1.log" Sep 10 19:11:20.855893 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.855879 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/0.log" Sep 10 19:11:20.855939 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.855914 2576 generic.go:356] "Generic (PLEG): container finished" podID="264d6101-b27d-4c83-bfe3-f70d9dcb0f74" containerID="59a9879a5e481ca59f8c8def598e4f0f337bc55cb2f302177656c62b7cac5805" exitCode=255 Sep 10 19:11:20.855976 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.855941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lq654" event={"ID":"264d6101-b27d-4c83-bfe3-f70d9dcb0f74","Type":"ContainerDied","Data":"59a9879a5e481ca59f8c8def598e4f0f337bc55cb2f302177656c62b7cac5805"} Sep 10 19:11:20.856031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.855994 2576 scope.go:117] "RemoveContainer" containerID="7cc0578ad0253c31fc53dd75ef03891bc4f1fd806d1144d9efa840eedc7b4d6e" Sep 10 19:11:20.856300 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.856278 2576 scope.go:117] "RemoveContainer" containerID="59a9879a5e481ca59f8c8def598e4f0f337bc55cb2f302177656c62b7cac5805" Sep 10 19:11:20.856525 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.856489 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-lq654_openshift-console-operator(264d6101-b27d-4c83-bfe3-f70d9dcb0f74)\"" pod="openshift-console-operator/console-operator-85b454874b-lq654" podUID="264d6101-b27d-4c83-bfe3-f70d9dcb0f74" Sep 10 19:11:20.883676 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.883637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:20.946528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.946493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:20.965736 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.965705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.965865 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:20.965757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:20.965865 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.965805 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:11:20.965983 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.965880 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:21.965858167 +0000 UTC m=+47.882514197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : configmap "kube-rbac-proxy" not found Sep 10 19:11:20.965983 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.965945 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:11:20.966096 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:20.965998 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:21.965979555 +0000 UTC m=+47.882635562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : secret "insights-runtime-extractor-tls" not found Sep 10 19:11:21.046678 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.046596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:21.233913 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.233876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:21.295452 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.295419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:21.516104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.516076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:21.517444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.517428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6qjq4_0f6c2cf3-35b5-4054-8b11-58f48bc41faf/dns-node-resolver/0.log" Sep 10 19:11:21.851360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.851329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:21.860697 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.860679 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/1.log" Sep 10 19:11:21.861023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.861007 2576 scope.go:117] "RemoveContainer" containerID="59a9879a5e481ca59f8c8def598e4f0f337bc55cb2f302177656c62b7cac5805" Sep 10 19:11:21.861197 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:21.861177 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-lq654_openshift-console-operator(264d6101-b27d-4c83-bfe3-f70d9dcb0f74)\"" pod="openshift-console-operator/console-operator-85b454874b-lq654" podUID="264d6101-b27d-4c83-bfe3-f70d9dcb0f74" Sep 10 19:11:21.975193 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.975151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:21.975357 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:21.975213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:21.975357 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:21.975226 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:11:21.975357 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:21.975297 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:23.975276802 +0000 UTC m=+49.891932811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : configmap "kube-rbac-proxy" not found Sep 10 19:11:21.975357 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:21.975323 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:11:21.975639 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:21.975364 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:23.975350407 +0000 UTC m=+49.892006434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : secret "insights-runtime-extractor-tls" not found Sep 10 19:11:22.518112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:22.518084 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2wk66_e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd/node-ca/0.log" Sep 10 19:11:23.132010 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:23.131977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:23.233034 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:23.233002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:23.522559 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:23.522488 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v4j2x_124cc65e-d0ed-4659-99d9-cb8a2c7a8763/serve-healthcheck-canary/0.log" Sep 10 19:11:23.597355 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:23.597322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:23.660366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:23.660333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:23.992886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:23.992857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:23.993065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:23.992897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:23.993065 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:23.992977 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:11:23.993065 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:23.992996 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:11:23.993065 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:23.993042 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:27.993026807 +0000 UTC m=+53.909682814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : configmap "kube-rbac-proxy" not found Sep 10 19:11:23.993065 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:23.993057 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:27.993050629 +0000 UTC m=+53.909706636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : secret "insights-runtime-extractor-tls" not found Sep 10 19:11:24.519590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:24.519556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:25.490551 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:25.490517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:25.807284 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:25.807256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:25.807645 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:25.807363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:25.807645 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.807445 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:25.807645 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.807502 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 19:11:25.807645 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.807518 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdb4b6644-2b49n: secret "image-registry-tls" not found Sep 10 19:11:25.807645 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.807511 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls podName:8d4d0730-72f3-4a8f-b169-b008b59ad4b0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:41.807493965 +0000 UTC m=+67.724149975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pnvfl" (UID: "8d4d0730-72f3-4a8f-b169-b008b59ad4b0") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:11:25.807645 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.807574 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls podName:6b526d36-86c1-4af9-a501-7be9605f0e5d nodeName:}" failed. No retries permitted until 2026-09-10 19:11:41.807561616 +0000 UTC m=+67.724217624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls") pod "image-registry-7bdb4b6644-2b49n" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d") : secret "image-registry-tls" not found Sep 10 19:11:25.908474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:25.908441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:25.908652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:25.908553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:25.908652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:25.908594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:25.908779 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.908595 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 19:11:25.908779 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.908661 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:41.908648154 +0000 UTC m=+67.825304162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : configmap references non-existent config key: service-ca.crt Sep 10 19:11:25.908779 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.908701 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:11:25.908779 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.908722 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls podName:06758258-e762-4934-9054-deda9c74df12 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:41.908705864 +0000 UTC m=+67.825361878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-26rsj" (UID: "06758258-e762-4934-9054-deda9c74df12") : secret "samples-operator-tls" not found Sep 10 19:11:25.908779 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:25.908743 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs podName:e01c9a3d-8c0f-482a-82fd-9e6d416f4647 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:41.908733525 +0000 UTC m=+67.825389540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs") pod "router-default-588979fc78-gdjfk" (UID: "e01c9a3d-8c0f-482a-82fd-9e6d416f4647") : secret "router-metrics-certs-default" not found Sep 10 19:11:26.009690 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:26.009654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:26.009839 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:26.009819 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:11:26.009896 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:26.009890 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls podName:d9ff1c36-92f9-4f39-82da-95672fcd5de9 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:42.009874668 +0000 UTC m=+67.926530675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls") pod "dns-default-t7gm6" (UID: "d9ff1c36-92f9-4f39-82da-95672fcd5de9") : secret "dns-default-metrics-tls" not found Sep 10 19:11:27.822215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:27.822176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:27.925528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:27.925494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:28.025665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.025629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:28.025772 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.025674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:28.025772 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:28.025766 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:11:28.025849 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:28.025821 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls podName:996a8504-cb25-471d-a86e-f34bd0f9915f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:36.025807375 +0000 UTC m=+61.942463383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whqpq" (UID: "996a8504-cb25-471d-a86e-f34bd0f9915f") : secret "insights-runtime-extractor-tls" not found Sep 10 19:11:28.026163 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.026144 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996a8504-cb25-471d-a86e-f34bd0f9915f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:28.094993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.094898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:28.230414 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.230356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:28.335804 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.335772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:28.445789 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.445719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:28.577723 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:28.577688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:29.532469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:29.532439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:29.738580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:29.738540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:29.908498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:29.908466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:30.310953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.310926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:30.387889 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.387860 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:30.388079 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.387920 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:30.388300 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.388282 2576 scope.go:117] "RemoveContainer" containerID="59a9879a5e481ca59f8c8def598e4f0f337bc55cb2f302177656c62b7cac5805" Sep 10 19:11:30.543108 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.543079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:30.890530 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.890505 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:11:30.890913 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.890899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/1.log" Sep 10 19:11:30.890962 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.890932 2576 generic.go:356] "Generic (PLEG): container finished" podID="264d6101-b27d-4c83-bfe3-f70d9dcb0f74" containerID="57ffc5c25bfa7c224244c0a9d4bda756599fd9c8941b797b7b1406ea2520727d" exitCode=255 Sep 10 19:11:30.891003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.890966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lq654" event={"ID":"264d6101-b27d-4c83-bfe3-f70d9dcb0f74","Type":"ContainerDied","Data":"57ffc5c25bfa7c224244c0a9d4bda756599fd9c8941b797b7b1406ea2520727d"} Sep 10 19:11:30.891037 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.891008 2576 scope.go:117] "RemoveContainer" containerID="59a9879a5e481ca59f8c8def598e4f0f337bc55cb2f302177656c62b7cac5805" Sep 10 19:11:30.891365 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:30.891346 2576 scope.go:117] "RemoveContainer" containerID="57ffc5c25bfa7c224244c0a9d4bda756599fd9c8941b797b7b1406ea2520727d" Sep 10 19:11:30.891544 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:30.891515 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-lq654_openshift-console-operator(264d6101-b27d-4c83-bfe3-f70d9dcb0f74)\"" pod="openshift-console-operator/console-operator-85b454874b-lq654" podUID="264d6101-b27d-4c83-bfe3-f70d9dcb0f74" Sep 10 19:11:31.127214 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:31.127164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:31.320234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:31.320206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:31.519447 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:31.519411 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:31.686830 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:31.686754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:31.849943 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:31.849912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:31.895221 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:31.895192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:11:31.951657 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:31.951591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:32.521211 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:32.521183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:32.616154 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:32.616121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:32.928213 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:32.928179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:33.164715 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:33.164685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:33.514706 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:33.514672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:34.126256 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:34.126226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:34.206052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:34.206022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:34.532897 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:34.532823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:34.765078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:34.765049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lfg" Sep 10 19:11:34.916571 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:34.916536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:35.139043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:35.139017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:35.706944 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:35.706909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:35.834065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:35.834035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:35.934848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:35.934820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:36.092826 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.092792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:36.095216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.095189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996a8504-cb25-471d-a86e-f34bd0f9915f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whqpq\" (UID: \"996a8504-cb25-471d-a86e-f34bd0f9915f\") " pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:36.112095 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.112071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:36.396552 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.396476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zl4hx\"" Sep 10 19:11:36.404439 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.404421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-whqpq" Sep 10 19:11:36.505838 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.505811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:36.573233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.573197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:36.676920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.676832 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-whqpq"] Sep 10 19:11:36.679511 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:36.679485 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996a8504_cb25_471d_a86e_f34bd0f9915f.slice/crio-8b2a51b6dd670e006d994f41ae7cd07063fe10e91fd67a020e8d77dd5eb65cec WatchSource:0}: Error finding container 8b2a51b6dd670e006d994f41ae7cd07063fe10e91fd67a020e8d77dd5eb65cec: Status 404 returned error can't find the container with id 8b2a51b6dd670e006d994f41ae7cd07063fe10e91fd67a020e8d77dd5eb65cec Sep 10 19:11:36.708652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.708629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:36.911949 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.911911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whqpq" event={"ID":"996a8504-cb25-471d-a86e-f34bd0f9915f","Type":"ContainerStarted","Data":"cecac2dea02bea7b360f1fb4214e21e9ab8792118436aab5149c46f1077261b5"} Sep 10 19:11:36.912105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:36.911962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whqpq" event={"ID":"996a8504-cb25-471d-a86e-f34bd0f9915f","Type":"ContainerStarted","Data":"8b2a51b6dd670e006d994f41ae7cd07063fe10e91fd67a020e8d77dd5eb65cec"} Sep 10 19:11:37.299250 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:37.299217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:37.802771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:37.802743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:37.919331 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:37.919283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whqpq" event={"ID":"996a8504-cb25-471d-a86e-f34bd0f9915f","Type":"ContainerStarted","Data":"6da87ca10a7dba96832924b2010a6ed4ae974d08ce78a9fc4f949ea5b0d12b87"} Sep 10 19:11:38.104501 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:38.104422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:38.306415 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:38.306365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:38.563388 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:38.563337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:39.928108 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:39.928071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whqpq" event={"ID":"996a8504-cb25-471d-a86e-f34bd0f9915f","Type":"ContainerStarted","Data":"fd705b5bf24092b21b6ffc66cf481952264f4783682705f2da7aa329f1677b4c"} Sep 10 19:11:39.947643 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:39.947597 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-whqpq" podStartSLOduration=17.73201004 podStartE2EDuration="19.947580661s" totalImagesPullingTime="2.215570621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:39.945701087 +0000 UTC m=+65.862357115" watchObservedRunningTime="2026-09-10 19:11:39.947580661 +0000 UTC m=+65.864236690" Sep 10 19:11:40.326586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.326546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:40.330122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.330100 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 19:11:40.339718 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.339692 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5421b-2ce5-469d-98e5-9fc2059de454-metrics-certs\") pod \"network-metrics-daemon-sr4z7\" (UID: \"04d5421b-2ce5-469d-98e5-9fc2059de454\") " pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:40.387650 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.387621 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:40.387758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.387681 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:40.388032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.388016 2576 scope.go:117] "RemoveContainer" containerID="57ffc5c25bfa7c224244c0a9d4bda756599fd9c8941b797b7b1406ea2520727d" Sep 10 19:11:40.388203 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:11:40.388182 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-lq654_openshift-console-operator(264d6101-b27d-4c83-bfe3-f70d9dcb0f74)\"" pod="openshift-console-operator/console-operator-85b454874b-lq654" podUID="264d6101-b27d-4c83-bfe3-f70d9dcb0f74" Sep 10 19:11:40.418593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.418573 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gdr\"" Sep 10 19:11:40.426431 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.426414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr4z7" Sep 10 19:11:40.427817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.427799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:40.428115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.428098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:40.430960 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.430944 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 19:11:40.431241 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.431223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqqs\" (UniqueName: \"kubernetes.io/projected/6a2c4140-61fe-483c-b71f-3d802b27a08f-kube-api-access-jzqqs\") pod \"network-check-target-4xrwb\" (UID: \"6a2c4140-61fe-483c-b71f-3d802b27a08f\") " pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:40.436316 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.436297 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zj5gq\"" Sep 10 19:11:40.440563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.440544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b14c0d55-d660-472c-ad93-f42bf3893fe9-original-pull-secret\") pod \"global-pull-secret-syncer-qxlb7\" (UID: \"b14c0d55-d660-472c-ad93-f42bf3893fe9\") " pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:40.444439 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.444420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:40.701177 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.701154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sr4z7"] Sep 10 19:11:40.703515 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:40.703493 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d5421b_2ce5_469d_98e5_9fc2059de454.slice/crio-4cf345abafb3fb8c834f356a6c7efb86ef4ec8710561e36ac937a6f5cd6bf4e1 WatchSource:0}: Error finding container 4cf345abafb3fb8c834f356a6c7efb86ef4ec8710561e36ac937a6f5cd6bf4e1: Status 404 returned error can't find the container with id 4cf345abafb3fb8c834f356a6c7efb86ef4ec8710561e36ac937a6f5cd6bf4e1 Sep 10 19:11:40.727026 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.727000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qxlb7" Sep 10 19:11:40.788518 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.788494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xrwb"] Sep 10 19:11:40.789547 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:40.789518 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2c4140_61fe_483c_b71f_3d802b27a08f.slice/crio-9f6c5453f0613dc5418ef50dd3e48d554bccfd776bcc78facb21a7af076eb9d5 WatchSource:0}: Error finding container 9f6c5453f0613dc5418ef50dd3e48d554bccfd776bcc78facb21a7af076eb9d5: Status 404 returned error can't find the container with id 9f6c5453f0613dc5418ef50dd3e48d554bccfd776bcc78facb21a7af076eb9d5 Sep 10 19:11:40.932808 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.932771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xrwb" event={"ID":"6a2c4140-61fe-483c-b71f-3d802b27a08f","Type":"ContainerStarted","Data":"e579d63ee07b22f3293aea51e74ce3617d42757dfdba2772956c51cab6e376ce"} Sep 10 19:11:40.933282 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.932831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xrwb" event={"ID":"6a2c4140-61fe-483c-b71f-3d802b27a08f","Type":"ContainerStarted","Data":"9f6c5453f0613dc5418ef50dd3e48d554bccfd776bcc78facb21a7af076eb9d5"} Sep 10 19:11:40.933282 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.932855 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:11:40.934054 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.934035 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sr4z7" event={"ID":"04d5421b-2ce5-469d-98e5-9fc2059de454","Type":"ContainerStarted","Data":"4cf345abafb3fb8c834f356a6c7efb86ef4ec8710561e36ac937a6f5cd6bf4e1"} Sep 10 19:11:40.955484 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.955409 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4xrwb" podStartSLOduration=66.955394386 podStartE2EDuration="1m6.955394386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:40.955012256 +0000 UTC m=+66.871668285" watchObservedRunningTime="2026-09-10 19:11:40.955394386 +0000 UTC m=+66.872050412" Sep 10 19:11:40.999524 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:40.999494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qxlb7"] Sep 10 19:11:41.002314 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:41.002286 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14c0d55_d660_472c_ad93_f42bf3893fe9.slice/crio-991c6aba35d68c0b73a0ca7c1fffe7f4aef86e4a5d87713e99c5af6dfe750290 WatchSource:0}: Error finding container 991c6aba35d68c0b73a0ca7c1fffe7f4aef86e4a5d87713e99c5af6dfe750290: Status 404 returned error can't find the container with id 991c6aba35d68c0b73a0ca7c1fffe7f4aef86e4a5d87713e99c5af6dfe750290 Sep 10 19:11:41.840865 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.840839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:41.840967 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.840909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:41.843992 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.843968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d4d0730-72f3-4a8f-b169-b008b59ad4b0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pnvfl\" (UID: \"8d4d0730-72f3-4a8f-b169-b008b59ad4b0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:41.844869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.844846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"image-registry-7bdb4b6644-2b49n\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:41.941071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.941005 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qxlb7" event={"ID":"b14c0d55-d660-472c-ad93-f42bf3893fe9","Type":"ContainerStarted","Data":"991c6aba35d68c0b73a0ca7c1fffe7f4aef86e4a5d87713e99c5af6dfe750290"} Sep 10 19:11:41.941482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.941307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:41.941482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.941364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:41.941482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.941419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:41.943061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.943034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-service-ca-bundle\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:41.943475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.943448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sr4z7" event={"ID":"04d5421b-2ce5-469d-98e5-9fc2059de454","Type":"ContainerStarted","Data":"cc703187d1dfcde828cf4e1d829aa60237c722794c417b97afb89fe49819c5ef"} Sep 10 19:11:41.943563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.943502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sr4z7" event={"ID":"04d5421b-2ce5-469d-98e5-9fc2059de454","Type":"ContainerStarted","Data":"a84eccaa065f1c7645166a8bd9909639dd0b579fdf88724293e98a1a93b4349b"} Sep 10 19:11:41.944813 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.944772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06758258-e762-4934-9054-deda9c74df12-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-26rsj\" (UID: \"06758258-e762-4934-9054-deda9c74df12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:41.944813 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.944808 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c9a3d-8c0f-482a-82fd-9e6d416f4647-metrics-certs\") pod \"router-default-588979fc78-gdjfk\" (UID: \"e01c9a3d-8c0f-482a-82fd-9e6d416f4647\") " pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:41.962471 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:41.962424 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sr4z7" podStartSLOduration=66.92458252 podStartE2EDuration="1m7.96240784s" totalImagesPullingTime="1.03782532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:41.961485047 +0000 UTC m=+67.878141074" watchObservedRunningTime="2026-09-10 19:11:41.96240784 +0000 UTC m=+67.879063868" Sep 10 19:11:42.041937 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.041907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:42.044736 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.044703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9ff1c36-92f9-4f39-82da-95672fcd5de9-metrics-tls\") pod \"dns-default-t7gm6\" (UID: \"d9ff1c36-92f9-4f39-82da-95672fcd5de9\") " pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:42.065327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.065297 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-99h89\"" Sep 10 19:11:42.073589 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.073564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" Sep 10 19:11:42.077481 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.077461 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8474t\"" Sep 10 19:11:42.085938 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.085687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:42.108590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.108567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wh5bc\"" Sep 10 19:11:42.116802 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.116771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:42.128938 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.128907 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gnzmg\"" Sep 10 19:11:42.138062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.137118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" Sep 10 19:11:42.255765 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.255736 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b59cd\"" Sep 10 19:11:42.263548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.263518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:42.367025 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.366948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl"] Sep 10 19:11:42.374214 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:42.374186 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4d0730_72f3_4a8f_b169_b008b59ad4b0.slice/crio-a609d57e92b29b8e6d9a63c7333e7363b196ea7cac645df77afef543e3c46a3c WatchSource:0}: Error finding container a609d57e92b29b8e6d9a63c7333e7363b196ea7cac645df77afef543e3c46a3c: Status 404 returned error can't find the container with id a609d57e92b29b8e6d9a63c7333e7363b196ea7cac645df77afef543e3c46a3c Sep 10 19:11:42.390109 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.390083 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bdb4b6644-2b49n"] Sep 10 19:11:42.394220 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:42.394191 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b526d36_86c1_4af9_a501_7be9605f0e5d.slice/crio-eb43f5dbe9b1459717d4e6db7a8137056b3f10cef4d1889b1153d82f250530db WatchSource:0}: Error finding container eb43f5dbe9b1459717d4e6db7a8137056b3f10cef4d1889b1153d82f250530db: Status 404 returned error can't find the container with id eb43f5dbe9b1459717d4e6db7a8137056b3f10cef4d1889b1153d82f250530db Sep 10 19:11:42.426851 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.426743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-588979fc78-gdjfk"] Sep 10 19:11:42.429493 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:42.429461 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01c9a3d_8c0f_482a_82fd_9e6d416f4647.slice/crio-4497fcc84c6865bc227078c854a1f705f1f9293d76f365eb9bd7d5574bd1cc22 WatchSource:0}: Error finding container 4497fcc84c6865bc227078c854a1f705f1f9293d76f365eb9bd7d5574bd1cc22: Status 404 returned error can't find the container with id 4497fcc84c6865bc227078c854a1f705f1f9293d76f365eb9bd7d5574bd1cc22 Sep 10 19:11:42.449205 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.449177 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj"] Sep 10 19:11:42.550577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.550549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t7gm6"] Sep 10 19:11:42.552966 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:11:42.552928 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ff1c36_92f9_4f39_82da_95672fcd5de9.slice/crio-13693d4cc002d6248e0566b21f143d38749d4067f38fa364972dbfed18ac79b8 WatchSource:0}: Error finding container 13693d4cc002d6248e0566b21f143d38749d4067f38fa364972dbfed18ac79b8: Status 404 returned error can't find the container with id 13693d4cc002d6248e0566b21f143d38749d4067f38fa364972dbfed18ac79b8 Sep 10 19:11:42.948727 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.948673 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7gm6" event={"ID":"d9ff1c36-92f9-4f39-82da-95672fcd5de9","Type":"ContainerStarted","Data":"13693d4cc002d6248e0566b21f143d38749d4067f38fa364972dbfed18ac79b8"} Sep 10 19:11:42.949870 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.949843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" event={"ID":"06758258-e762-4934-9054-deda9c74df12","Type":"ContainerStarted","Data":"a9fa56c8ee1b2bce18305a48949539c05d0d4601653400b4b579151e911b409e"} Sep 10 19:11:42.951696 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.951674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" event={"ID":"6b526d36-86c1-4af9-a501-7be9605f0e5d","Type":"ContainerStarted","Data":"6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9"} Sep 10 19:11:42.951812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.951716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" event={"ID":"6b526d36-86c1-4af9-a501-7be9605f0e5d","Type":"ContainerStarted","Data":"eb43f5dbe9b1459717d4e6db7a8137056b3f10cef4d1889b1153d82f250530db"} Sep 10 19:11:42.951872 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.951815 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:11:42.953213 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.953188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" event={"ID":"8d4d0730-72f3-4a8f-b169-b008b59ad4b0","Type":"ContainerStarted","Data":"a609d57e92b29b8e6d9a63c7333e7363b196ea7cac645df77afef543e3c46a3c"} Sep 10 19:11:42.954766 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.954741 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-588979fc78-gdjfk" event={"ID":"e01c9a3d-8c0f-482a-82fd-9e6d416f4647","Type":"ContainerStarted","Data":"9256d0341e6f08a5a43465aa2454757278c2bb25d4a1c76ced4ac78ca0d9678f"} Sep 10 19:11:42.954858 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.954786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-588979fc78-gdjfk" event={"ID":"e01c9a3d-8c0f-482a-82fd-9e6d416f4647","Type":"ContainerStarted","Data":"4497fcc84c6865bc227078c854a1f705f1f9293d76f365eb9bd7d5574bd1cc22"} Sep 10 19:11:42.979718 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:42.979674 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" podStartSLOduration=67.979660132 podStartE2EDuration="1m7.979660132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:42.97940332 +0000 UTC m=+68.896059342" watchObservedRunningTime="2026-09-10 19:11:42.979660132 +0000 UTC m=+68.896316155" Sep 10 19:11:43.117544 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:43.117504 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:43.120317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:43.120293 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:43.143274 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:43.143225 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-588979fc78-gdjfk" podStartSLOduration=62.143209849 podStartE2EDuration="1m2.143209849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:43.007345823 +0000 UTC m=+68.924001851" watchObservedRunningTime="2026-09-10 19:11:43.143209849 +0000 UTC m=+69.059865877" Sep 10 19:11:43.961246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:43.961151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:43.962780 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:43.962756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-588979fc78-gdjfk" Sep 10 19:11:45.262056 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:45.262022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:46.710043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.710001 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz"] Sep 10 19:11:46.755246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.755205 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz"] Sep 10 19:11:46.755443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.755342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.765811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.765549 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 19:11:46.765811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.765620 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 19:11:46.765811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.765560 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 19:11:46.766636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.766105 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 19:11:46.844935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.844899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:46.885038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.885004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-klusterlet-config\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.885038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.885041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txqz\" (UniqueName: \"kubernetes.io/projected/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-kube-api-access-8txqz\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.885265 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.885083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-tmp\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.976851 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.976778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:46.985739 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.985715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-klusterlet-config\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.985869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.985752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txqz\" (UniqueName: \"kubernetes.io/projected/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-kube-api-access-8txqz\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.985869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.985801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-tmp\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.986184 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.986164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-tmp\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.988197 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.988173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-klusterlet-config\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:46.995232 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:46.995207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txqz\" (UniqueName: \"kubernetes.io/projected/8ce4970f-cab5-45fb-84b3-d11f86fcc95c-kube-api-access-8txqz\") pod \"klusterlet-addon-workmgr-59fdf99bbf-nwtkz\" (UID: \"8ce4970f-cab5-45fb-84b3-d11f86fcc95c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:47.069389 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:47.069337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:47.080255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:47.080224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:47.372027 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:47.371994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:47.431194 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:47.431146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:48.372937 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.372872 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz"] Sep 10 19:11:48.580209 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.580100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:48.672166 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.672131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:48.757580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.757552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:48.778502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.778470 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks"] Sep 10 19:11:48.796614 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.796590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" Sep 10 19:11:48.797344 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.797322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks"] Sep 10 19:11:48.799223 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.799202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 19:11:48.799324 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.799249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6tf4s\"" Sep 10 19:11:48.821750 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.821726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:48.886765 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.886697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:48.899930 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.899907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9c3342ec-124f-40a3-b8f9-9b4d945e380d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8npks\" (UID: \"9c3342ec-124f-40a3-b8f9-9b4d945e380d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" Sep 10 19:11:48.974691 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.974666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:48.979493 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.979466 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qxlb7" event={"ID":"b14c0d55-d660-472c-ad93-f42bf3893fe9","Type":"ContainerStarted","Data":"e432a360a7b6348e526858429085c8addb4bdfad6ed42a491ec415f8be116eeb"} Sep 10 19:11:48.981431 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.981399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7gm6" event={"ID":"d9ff1c36-92f9-4f39-82da-95672fcd5de9","Type":"ContainerStarted","Data":"69f04164e81c14bb81046045c77797135ac1351ab474c197526f341a7eef9c8e"} Sep 10 19:11:48.981557 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.981450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7gm6" event={"ID":"d9ff1c36-92f9-4f39-82da-95672fcd5de9","Type":"ContainerStarted","Data":"20cf17786dbbc73ea550d68df4fe4a280250243a5c90e6aa37ddba3897b4e51e"} Sep 10 19:11:48.981557 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.981503 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:48.983208 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.983181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" event={"ID":"06758258-e762-4934-9054-deda9c74df12","Type":"ContainerStarted","Data":"05324855a7d744c948b5632f57b712ddf0db2d7d9a4ec27aa367b20518f9ba12"} Sep 10 19:11:48.983316 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.983238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" event={"ID":"06758258-e762-4934-9054-deda9c74df12","Type":"ContainerStarted","Data":"e6b1547f3f6a8b7b34684c752d02154cf1236400493217b0a2a11d99715fe813"} Sep 10 19:11:48.984759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.984737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" event={"ID":"8d4d0730-72f3-4a8f-b169-b008b59ad4b0","Type":"ContainerStarted","Data":"9ac1768cae26602fcf302d611e3cbca3eae478794bc5c2821013cf49bdf26be4"} Sep 10 19:11:48.985882 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:48.985864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" event={"ID":"8ce4970f-cab5-45fb-84b3-d11f86fcc95c","Type":"ContainerStarted","Data":"e21bed14cd16f3d1890bbc283cad2494112ca5cde3439545d8a170debc9fc216"} Sep 10 19:11:49.000393 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.000308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9c3342ec-124f-40a3-b8f9-9b4d945e380d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8npks\" (UID: \"9c3342ec-124f-40a3-b8f9-9b4d945e380d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" Sep 10 19:11:49.000720 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.000681 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qxlb7" podStartSLOduration=66.897256729 podStartE2EDuration="1m14.000667961s" totalImagesPullingTime="7.103411232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:49.000173585 +0000 UTC m=+74.916829635" watchObservedRunningTime="2026-09-10 19:11:49.000667961 +0000 UTC m=+74.917323991" Sep 10 19:11:49.002831 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.002809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9c3342ec-124f-40a3-b8f9-9b4d945e380d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8npks\" (UID: \"9c3342ec-124f-40a3-b8f9-9b4d945e380d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" Sep 10 19:11:49.023758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.023717 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t7gm6" podStartSLOduration=34.470304395 podStartE2EDuration="40.023703689s" totalImagesPullingTime="5.553399294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:49.021961171 +0000 UTC m=+74.938617199" watchObservedRunningTime="2026-09-10 19:11:49.023703689 +0000 UTC m=+74.940359717" Sep 10 19:11:49.049571 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.049528 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pnvfl" podStartSLOduration=62.319368724 podStartE2EDuration="1m8.049513603s" totalImagesPullingTime="5.730144879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:49.048398353 +0000 UTC m=+74.965054378" watchObservedRunningTime="2026-09-10 19:11:49.049513603 +0000 UTC m=+74.966169632" Sep 10 19:11:49.105992 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.105970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" Sep 10 19:11:49.393737 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.393553 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-26rsj" podStartSLOduration=61.774594552 podStartE2EDuration="1m7.393534123s" totalImagesPullingTime="5.618939571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:49.090117167 +0000 UTC m=+75.006773196" watchObservedRunningTime="2026-09-10 19:11:49.393534123 +0000 UTC m=+75.310190155" Sep 10 19:11:49.394338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.394308 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks"] Sep 10 19:11:49.998003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:49.997960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" event={"ID":"9c3342ec-124f-40a3-b8f9-9b4d945e380d","Type":"ContainerStarted","Data":"0efee2cf70e54e4280baaf52d6c43d22a884ae2209c3a8c33a3a329719c87f33"} Sep 10 19:11:50.853310 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:50.853265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:50.922929 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:50.922895 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:51.871384 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:51.871326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:51.929584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:51.929546 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:53.596817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:53.596786 2576 scope.go:117] "RemoveContainer" containerID="57ffc5c25bfa7c224244c0a9d4bda756599fd9c8941b797b7b1406ea2520727d" Sep 10 19:11:53.846784 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:53.846732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:53.912665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:53.912571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:54.014274 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.014240 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" event={"ID":"9c3342ec-124f-40a3-b8f9-9b4d945e380d","Type":"ContainerStarted","Data":"ae5f79ccc744c9c421b0dfa4b6db486aa37ecc05448bccdc0fa4188cd4f2ccad"} Sep 10 19:11:54.014475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.014296 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" Sep 10 19:11:54.016017 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.015990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" event={"ID":"8ce4970f-cab5-45fb-84b3-d11f86fcc95c","Type":"ContainerStarted","Data":"9b9fc09e0dc37686820f6d55681cdaf52df23916be32f55af81340ec19edfaf2"} Sep 10 19:11:54.016172 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.016151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:54.017809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.017790 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" Sep 10 19:11:54.018317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.018300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:11:54.018453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.018359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lq654" event={"ID":"264d6101-b27d-4c83-bfe3-f70d9dcb0f74","Type":"ContainerStarted","Data":"2f0590cd635c4630fcceb6bbc717a8ec36bf04c64441f746e5d691285f3673e2"} Sep 10 19:11:54.018724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.018677 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:54.019993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.019974 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" Sep 10 19:11:54.031853 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.031808 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8npks" podStartSLOduration=2.144729269 podStartE2EDuration="6.031791065s" totalImagesPullingTime="3.887061796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:54.03109929 +0000 UTC m=+79.947755319" watchObservedRunningTime="2026-09-10 19:11:54.031791065 +0000 UTC m=+79.948447096" Sep 10 19:11:54.049525 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.049484 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-lq654" podStartSLOduration=61.951101037 podStartE2EDuration="1m10.049472326s" totalImagesPullingTime="8.098371289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:54.048817558 +0000 UTC m=+79.965473587" watchObservedRunningTime="2026-09-10 19:11:54.049472326 +0000 UTC m=+79.966128355" Sep 10 19:11:54.079995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.079941 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59fdf99bbf-nwtkz" podStartSLOduration=3.179772149 podStartE2EDuration="8.079923039s" totalImagesPullingTime="4.90015089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:54.07972995 +0000 UTC m=+79.996385980" watchObservedRunningTime="2026-09-10 19:11:54.079923039 +0000 UTC m=+79.996579070" Sep 10 19:11:54.171431 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.171338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-lq654" Sep 10 19:11:54.362085 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.362048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:54.462161 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.462086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:54.518130 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.518099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:54.805443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.805409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:54.883007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.882974 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:54.985181 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:54.985148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:55.158543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:55.158456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:55.260278 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:55.260246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:55.336905 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:55.336878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:55.425081 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:55.425005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:55.525888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:55.525861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:55.841026 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:55.840987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:55.996939 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:55.996802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:56.095097 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:56.095021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:56.260860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:56.260828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:56.361444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:56.361341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:56.462778 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:56.462742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:57.252509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:57.252470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:57.413615 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:57.413589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:58.823993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:58.823960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:58.896477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:58.896444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:58.984941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:58.984912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:59.001154 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:59.001133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t7gm6" Sep 10 19:11:59.080131 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:59.080058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:11:59.965042 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:11:59.965010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:00.178582 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.178545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:00.230341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.230262 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7zj7q"] Sep 10 19:12:00.235049 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.235028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.237359 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.237338 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 19:12:00.237651 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.237637 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 19:12:00.237936 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.237921 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 19:12:00.238296 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.238284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4fnns\"" Sep 10 19:12:00.239116 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.239099 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 19:12:00.280702 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.280677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:00.391730 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.391690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:00.395034 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-tls\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5hf\" (UniqueName: \"kubernetes.io/projected/fcba22d5-19ed-435d-8f74-dad0a96395c0-kube-api-access-wb5hf\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-sys\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-root\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-wtmp\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-textfile\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.395473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.395301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcba22d5-19ed-435d-8f74-dad0a96395c0-metrics-client-ca\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.491487 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.491414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:00.496574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-wtmp\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-textfile\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcba22d5-19ed-435d-8f74-dad0a96395c0-metrics-client-ca\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-tls\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-wtmp\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5hf\" (UniqueName: \"kubernetes.io/projected/fcba22d5-19ed-435d-8f74-dad0a96395c0-kube-api-access-wb5hf\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-sys\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.496869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-root\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.497155 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.496930 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-root\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.497155 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.497032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-textfile\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.497155 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.497095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcba22d5-19ed-435d-8f74-dad0a96395c0-sys\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.497308 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.497283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.497365 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.497345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcba22d5-19ed-435d-8f74-dad0a96395c0-metrics-client-ca\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.499260 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.499238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.499636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.499622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcba22d5-19ed-435d-8f74-dad0a96395c0-node-exporter-tls\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.505066 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.505046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5hf\" (UniqueName: \"kubernetes.io/projected/fcba22d5-19ed-435d-8f74-dad0a96395c0-kube-api-access-wb5hf\") pod \"node-exporter-7zj7q\" (UID: \"fcba22d5-19ed-435d-8f74-dad0a96395c0\") " pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.544836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.544811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7zj7q" Sep 10 19:12:00.557448 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:12:00.557420 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcba22d5_19ed_435d_8f74_dad0a96395c0.slice/crio-c48d52bfcf47a4c843dbf102ef4ff80749c4af967fe56a5b42f76933a657604e WatchSource:0}: Error finding container c48d52bfcf47a4c843dbf102ef4ff80749c4af967fe56a5b42f76933a657604e: Status 404 returned error can't find the container with id c48d52bfcf47a4c843dbf102ef4ff80749c4af967fe56a5b42f76933a657604e Sep 10 19:12:00.591625 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.591594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:00.691300 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:00.691272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:01.045773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:01.045737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7zj7q" event={"ID":"fcba22d5-19ed-435d-8f74-dad0a96395c0","Type":"ContainerStarted","Data":"c48d52bfcf47a4c843dbf102ef4ff80749c4af967fe56a5b42f76933a657604e"} Sep 10 19:12:01.205522 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:01.205469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:01.308005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:01.307928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:01.404510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:01.404478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:01.658568 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:01.658459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:02.055479 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:02.055439 2576 generic.go:356] "Generic (PLEG): container finished" podID="fcba22d5-19ed-435d-8f74-dad0a96395c0" containerID="00f266d2c04a24a845356216189e042d84a9654da3252f214805b09378a5e4fb" exitCode=0 Sep 10 19:12:02.056010 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:02.055513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7zj7q" event={"ID":"fcba22d5-19ed-435d-8f74-dad0a96395c0","Type":"ContainerDied","Data":"00f266d2c04a24a845356216189e042d84a9654da3252f214805b09378a5e4fb"} Sep 10 19:12:02.090924 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:02.090893 2576 patch_prober.go:28] interesting pod/image-registry-7bdb4b6644-2b49n container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 19:12:02.091030 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:02.090946 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" podUID="6b526d36-86c1-4af9-a501-7be9605f0e5d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 19:12:03.064114 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.064070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7zj7q" event={"ID":"fcba22d5-19ed-435d-8f74-dad0a96395c0","Type":"ContainerStarted","Data":"6ab4c64b2c81120d00846c6af6b7ae6c72b5aad8c6493f63b138d27fd7566373"} Sep 10 19:12:03.064543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.064131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7zj7q" event={"ID":"fcba22d5-19ed-435d-8f74-dad0a96395c0","Type":"ContainerStarted","Data":"56b77b0242bd78a2892032f54410b67c5526312319394f49624ea96798d8ab92"} Sep 10 19:12:03.070011 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.069981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:03.083614 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.083565 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7zj7q" podStartSLOduration=1.7860970200000001 podStartE2EDuration="3.083546423s" totalImagesPullingTime="1.229434311s" totalInitContainerRuntime="68.015092ms" isStatefulPod=true podCreationTimestamp="2026-09-10 19:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:03.082239788 +0000 UTC m=+88.998895817" watchObservedRunningTime="2026-09-10 19:12:03.083546423 +0000 UTC m=+89.000202452" Sep 10 19:12:03.181097 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.181062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:03.301043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.300992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:03.889070 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.889040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:03.966435 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.966408 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:12:03.991662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:03.991632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:04.258098 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:04.258028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:04.359266 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:04.359229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:04.458762 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:04.458729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:04.755525 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:04.755489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:04.852814 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:04.852774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:04.969546 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:04.969514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:05.032523 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:05.032425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:05.372335 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:05.372306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:06.622394 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:06.622338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:07.257113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:07.257086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:07.357446 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:07.357416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:10.695983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:10.695951 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bdb4b6644-2b49n"] Sep 10 19:12:10.802719 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:10.802686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:10.901968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:10.901939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:10.969480 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:10.969361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:11.579186 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:11.579134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:11.680471 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:11.680444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:11.736282 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:11.736234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:11.946140 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:11.946059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4xrwb" Sep 10 19:12:12.068563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.068530 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55bc79dbd-gdkhv"] Sep 10 19:12:12.074341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.074320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.077007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.076980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 19:12:12.077007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.076995 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 19:12:12.077327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.077306 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fnqjb\"" Sep 10 19:12:12.077424 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.077365 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 19:12:12.077482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.077364 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 19:12:12.077529 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.077479 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 19:12:12.077657 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.077640 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 19:12:12.077728 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.077714 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 19:12:12.082209 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.082192 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 19:12:12.082923 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.082905 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bc79dbd-gdkhv"] Sep 10 19:12:12.186327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.186291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-service-ca\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.186504 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.186333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-oauth-config\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.186504 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.186411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-oauth-serving-cert\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.186504 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.186460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-serving-cert\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.186504 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.186481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-console-config\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.186636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.186529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-trusted-ca-bundle\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.186636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.186546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crw9\" (UniqueName: \"kubernetes.io/projected/73e5d206-2a90-4446-9eb5-cabfda46877f-kube-api-access-5crw9\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.287247 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.287216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-oauth-serving-cert\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.287419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.287267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-serving-cert\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.287419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.287402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-console-config\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.287498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.287485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-trusted-ca-bundle\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.287531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.287509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crw9\" (UniqueName: \"kubernetes.io/projected/73e5d206-2a90-4446-9eb5-cabfda46877f-kube-api-access-5crw9\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.287577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.287561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-service-ca\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.287773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.287698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-oauth-config\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.288141 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.288117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-oauth-serving-cert\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.288441 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.288147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-service-ca\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.288441 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.288118 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-console-config\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.288441 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.288408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-trusted-ca-bundle\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.289950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.289925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-serving-cert\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.290141 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.290125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-oauth-config\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.296461 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.296440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crw9\" (UniqueName: \"kubernetes.io/projected/73e5d206-2a90-4446-9eb5-cabfda46877f-kube-api-access-5crw9\") pod \"console-55bc79dbd-gdkhv\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.384890 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.384848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:12.659726 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:12.659654 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bc79dbd-gdkhv"] Sep 10 19:12:12.662459 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:12:12.662435 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e5d206_2a90_4446_9eb5_cabfda46877f.slice/crio-75a75cc6442b8d32ac121a36c918dc070ab8c1f9fd0d7db5da3884829bdaa6b8 WatchSource:0}: Error finding container 75a75cc6442b8d32ac121a36c918dc070ab8c1f9fd0d7db5da3884829bdaa6b8: Status 404 returned error can't find the container with id 75a75cc6442b8d32ac121a36c918dc070ab8c1f9fd0d7db5da3884829bdaa6b8 Sep 10 19:12:13.105386 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:13.105347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bc79dbd-gdkhv" event={"ID":"73e5d206-2a90-4446-9eb5-cabfda46877f","Type":"ContainerStarted","Data":"75a75cc6442b8d32ac121a36c918dc070ab8c1f9fd0d7db5da3884829bdaa6b8"} Sep 10 19:12:13.220214 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:13.220183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:13.284903 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:13.284873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:13.787821 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:13.787777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:13.853851 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:13.853817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:16.119017 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:16.118932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bc79dbd-gdkhv" event={"ID":"73e5d206-2a90-4446-9eb5-cabfda46877f","Type":"ContainerStarted","Data":"1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42"} Sep 10 19:12:16.139698 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:16.139646 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55bc79dbd-gdkhv" podStartSLOduration=0.950582887 podStartE2EDuration="4.139629755s" totalImagesPullingTime="3.189046868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:16.138786343 +0000 UTC m=+102.055442372" watchObservedRunningTime="2026-09-10 19:12:16.139629755 +0000 UTC m=+102.056285856" Sep 10 19:12:17.073667 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:17.073639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:17.238481 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:17.238450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:20.634498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.634464 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65f7df74b5-zx5bg"] Sep 10 19:12:20.639311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.639290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.649463 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.649440 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f7df74b5-zx5bg"] Sep 10 19:12:20.658455 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.658425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-oauth-config\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.658559 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.658482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-service-ca\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.658602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.658552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-trusted-ca-bundle\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.658635 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.658606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-oauth-serving-cert\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.658668 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.658641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzwph\" (UniqueName: \"kubernetes.io/projected/5a057a7a-165a-4efd-9804-6ea1b5b289af-kube-api-access-nzwph\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.658711 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.658696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-serving-cert\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.658785 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.658767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-config\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.676642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.676604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:20.759572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.759542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-serving-cert\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.759740 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.759595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-config\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.759740 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.759613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-oauth-config\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.759740 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.759638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-service-ca\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.759897 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.759831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-trusted-ca-bundle\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.759897 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.759891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-oauth-serving-cert\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.760051 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.760026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzwph\" (UniqueName: \"kubernetes.io/projected/5a057a7a-165a-4efd-9804-6ea1b5b289af-kube-api-access-nzwph\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.760498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.760465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-service-ca\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.760603 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.760530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-config\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.760603 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.760530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-oauth-serving-cert\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.760725 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.760654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-trusted-ca-bundle\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.762152 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.762132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-serving-cert\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.762262 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.762158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-oauth-config\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.768508 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.768487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzwph\" (UniqueName: \"kubernetes.io/projected/5a057a7a-165a-4efd-9804-6ea1b5b289af-kube-api-access-nzwph\") pod \"console-65f7df74b5-zx5bg\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:20.780865 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.780842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:20.878164 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.878126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:20.946482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.946363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:20.948435 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:20.948418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:21.231581 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:21.231462 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f7df74b5-zx5bg"] Sep 10 19:12:21.235459 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:12:21.235433 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a057a7a_165a_4efd_9804_6ea1b5b289af.slice/crio-5051a3b431c6d16b2e89a0930b78905d1e33c30ccabca60a1bc71e6fe5b3b2ae WatchSource:0}: Error finding container 5051a3b431c6d16b2e89a0930b78905d1e33c30ccabca60a1bc71e6fe5b3b2ae: Status 404 returned error can't find the container with id 5051a3b431c6d16b2e89a0930b78905d1e33c30ccabca60a1bc71e6fe5b3b2ae Sep 10 19:12:22.146027 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.145995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f7df74b5-zx5bg" event={"ID":"5a057a7a-165a-4efd-9804-6ea1b5b289af","Type":"ContainerStarted","Data":"883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7"} Sep 10 19:12:22.146436 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.146048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f7df74b5-zx5bg" event={"ID":"5a057a7a-165a-4efd-9804-6ea1b5b289af","Type":"ContainerStarted","Data":"5051a3b431c6d16b2e89a0930b78905d1e33c30ccabca60a1bc71e6fe5b3b2ae"} Sep 10 19:12:22.166431 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.166363 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65f7df74b5-zx5bg" podStartSLOduration=2.166344512 podStartE2EDuration="2.166344512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:22.164969481 +0000 UTC m=+108.081625510" watchObservedRunningTime="2026-09-10 19:12:22.166344512 +0000 UTC m=+108.083000543" Sep 10 19:12:22.385191 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.385150 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:22.385363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.385257 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:22.390023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.389999 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:22.470664 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.470578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:22.538592 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:22.538555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:23.153761 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:23.153732 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:23.742494 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:23.742463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:23.843958 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:23.843923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:24.302689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:24.302650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:24.403758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:24.403720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:25.702866 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:25.702838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:25.803794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:25.803763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:30.178175 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:30.178131 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec9ceda2-9fc5-439b-86c2-171a6effa80d" containerID="5a922f20286c7caa9d2a743cdc033e2138d96db8d2f58e13030a4671e0be3786" exitCode=0 Sep 10 19:12:30.178583 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:30.178186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" event={"ID":"ec9ceda2-9fc5-439b-86c2-171a6effa80d","Type":"ContainerDied","Data":"5a922f20286c7caa9d2a743cdc033e2138d96db8d2f58e13030a4671e0be3786"} Sep 10 19:12:30.178583 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:30.178552 2576 scope.go:117] "RemoveContainer" containerID="5a922f20286c7caa9d2a743cdc033e2138d96db8d2f58e13030a4671e0be3786" Sep 10 19:12:30.949239 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:30.949201 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:30.949436 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:30.949262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:30.954106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:30.954085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:31.184839 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:31.184807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-slhxt" event={"ID":"ec9ceda2-9fc5-439b-86c2-171a6effa80d","Type":"ContainerStarted","Data":"cfbd48f6f9496f6e6c9538be13b2a1a8d3d2065eceb687e6b5ae1e185e66a395"} Sep 10 19:12:31.188781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:31.188758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:12:31.256228 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:31.256153 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bc79dbd-gdkhv"] Sep 10 19:12:32.317716 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:32.317678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:32.420068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:32.420035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:35.716135 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:35.716088 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" podUID="6b526d36-86c1-4af9-a501-7be9605f0e5d" containerName="registry" containerID="cri-o://6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9" gracePeriod=30 Sep 10 19:12:35.836868 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:35.836841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:35.908023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:35.907935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:12:35.961143 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:35.961117 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:12:36.085637 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085597 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-trusted-ca\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.085828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085647 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-installation-pull-secrets\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.085828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085687 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.085828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkzqh\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-kube-api-access-rkzqh\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.085828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085747 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-bound-sa-token\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.085828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085805 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-image-registry-private-configuration\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.086081 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-certificates\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.086081 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.085877 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b526d36-86c1-4af9-a501-7be9605f0e5d-ca-trust-extracted\") pod \"6b526d36-86c1-4af9-a501-7be9605f0e5d\" (UID: \"6b526d36-86c1-4af9-a501-7be9605f0e5d\") " Sep 10 19:12:36.086179 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.086101 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:12:36.086825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.086769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:12:36.088550 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.088515 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:12:36.088697 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.088632 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:12:36.088771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.088746 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:12:36.088828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.088781 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-kube-api-access-rkzqh" (OuterVolumeSpecName: "kube-api-access-rkzqh") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "kube-api-access-rkzqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:12:36.088881 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.088839 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:12:36.093804 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.093781 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b526d36-86c1-4af9-a501-7be9605f0e5d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6b526d36-86c1-4af9-a501-7be9605f0e5d" (UID: "6b526d36-86c1-4af9-a501-7be9605f0e5d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:12:36.186986 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.186956 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-image-registry-private-configuration\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.186986 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.186984 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-certificates\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.186986 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.186993 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b526d36-86c1-4af9-a501-7be9605f0e5d-ca-trust-extracted\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.187216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.187002 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b526d36-86c1-4af9-a501-7be9605f0e5d-trusted-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.187216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.187011 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b526d36-86c1-4af9-a501-7be9605f0e5d-installation-pull-secrets\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.187216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.187019 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-registry-tls\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.187216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.187026 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkzqh\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-kube-api-access-rkzqh\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.187216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.187035 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b526d36-86c1-4af9-a501-7be9605f0e5d-bound-sa-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:36.205289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.205262 2576 generic.go:356] "Generic (PLEG): container finished" podID="6b526d36-86c1-4af9-a501-7be9605f0e5d" containerID="6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9" exitCode=0 Sep 10 19:12:36.205459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.205305 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" Sep 10 19:12:36.205459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.205357 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" event={"ID":"6b526d36-86c1-4af9-a501-7be9605f0e5d","Type":"ContainerDied","Data":"6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9"} Sep 10 19:12:36.205527 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.205458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdb4b6644-2b49n" event={"ID":"6b526d36-86c1-4af9-a501-7be9605f0e5d","Type":"ContainerDied","Data":"eb43f5dbe9b1459717d4e6db7a8137056b3f10cef4d1889b1153d82f250530db"} Sep 10 19:12:36.205527 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.205483 2576 scope.go:117] "RemoveContainer" containerID="6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9" Sep 10 19:12:36.214799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.214782 2576 scope.go:117] "RemoveContainer" containerID="6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9" Sep 10 19:12:36.215077 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:12:36.215058 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9\": container with ID starting with 6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9 not found: ID does not exist" containerID="6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9" Sep 10 19:12:36.215121 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.215086 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9"} err="failed to get container status \"6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9\": rpc error: code = NotFound desc = could not find container \"6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9\": container with ID starting with 6e2f56a28e4655acd51ba396dc6fc8de5f37cadf6da74090283ad1dba55f48d9 not found: ID does not exist" Sep 10 19:12:36.226732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.226708 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bdb4b6644-2b49n"] Sep 10 19:12:36.229642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.229617 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bdb4b6644-2b49n"] Sep 10 19:12:36.601138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:36.601106 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b526d36-86c1-4af9-a501-7be9605f0e5d" path="/var/lib/kubelet/pods/6b526d36-86c1-4af9-a501-7be9605f0e5d/volumes" Sep 10 19:12:40.221493 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:40.221459 2576 generic.go:356] "Generic (PLEG): container finished" podID="226486d2-fe60-4143-9151-43c5cae901ee" containerID="d7050974e0cae1d9e845808b723ce80235d521ef6458cfcbd9f3da363113ee85" exitCode=0 Sep 10 19:12:40.221888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:40.221533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" event={"ID":"226486d2-fe60-4143-9151-43c5cae901ee","Type":"ContainerDied","Data":"d7050974e0cae1d9e845808b723ce80235d521ef6458cfcbd9f3da363113ee85"} Sep 10 19:12:40.221888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:40.221824 2576 scope.go:117] "RemoveContainer" containerID="d7050974e0cae1d9e845808b723ce80235d521ef6458cfcbd9f3da363113ee85" Sep 10 19:12:41.227184 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:41.227152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwbxn" event={"ID":"226486d2-fe60-4143-9151-43c5cae901ee","Type":"ContainerStarted","Data":"0eeb4b546c1dae80176069b8b744908acc5c24ed772e2e7814097f7324148b9a"} Sep 10 19:12:50.261215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:50.261177 2576 generic.go:356] "Generic (PLEG): container finished" podID="cdc51729-75fb-49a9-a31e-827cde63ffc1" containerID="4b8f96afcee4d8923aee51d91750f6bea796e35145349139443ff29e66220a86" exitCode=0 Sep 10 19:12:50.261656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:50.261253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" event={"ID":"cdc51729-75fb-49a9-a31e-827cde63ffc1","Type":"ContainerDied","Data":"4b8f96afcee4d8923aee51d91750f6bea796e35145349139443ff29e66220a86"} Sep 10 19:12:50.261656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:50.261631 2576 scope.go:117] "RemoveContainer" containerID="4b8f96afcee4d8923aee51d91750f6bea796e35145349139443ff29e66220a86" Sep 10 19:12:51.267446 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:51.267403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zrw7b" event={"ID":"cdc51729-75fb-49a9-a31e-827cde63ffc1","Type":"ContainerStarted","Data":"01e42ea0ebad99550dacfa1002f603227ae1e6e69186d707801b0fc10101ef90"} Sep 10 19:12:56.279475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.279429 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55bc79dbd-gdkhv" podUID="73e5d206-2a90-4446-9eb5-cabfda46877f" containerName="console" containerID="cri-o://1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42" gracePeriod=15 Sep 10 19:12:56.521663 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.521638 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:56.651986 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.651954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-oauth-serving-cert\") pod \"73e5d206-2a90-4446-9eb5-cabfda46877f\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " Sep 10 19:12:56.652149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.651997 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-service-ca\") pod \"73e5d206-2a90-4446-9eb5-cabfda46877f\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " Sep 10 19:12:56.652149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652023 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-oauth-config\") pod \"73e5d206-2a90-4446-9eb5-cabfda46877f\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " Sep 10 19:12:56.652149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-serving-cert\") pod \"73e5d206-2a90-4446-9eb5-cabfda46877f\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " Sep 10 19:12:56.652149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652071 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-console-config\") pod \"73e5d206-2a90-4446-9eb5-cabfda46877f\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " Sep 10 19:12:56.652149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652093 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-trusted-ca-bundle\") pod \"73e5d206-2a90-4446-9eb5-cabfda46877f\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " Sep 10 19:12:56.652149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652115 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5crw9\" (UniqueName: \"kubernetes.io/projected/73e5d206-2a90-4446-9eb5-cabfda46877f-kube-api-access-5crw9\") pod \"73e5d206-2a90-4446-9eb5-cabfda46877f\" (UID: \"73e5d206-2a90-4446-9eb5-cabfda46877f\") " Sep 10 19:12:56.652462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652354 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73e5d206-2a90-4446-9eb5-cabfda46877f" (UID: "73e5d206-2a90-4446-9eb5-cabfda46877f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:12:56.652659 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652616 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-console-config" (OuterVolumeSpecName: "console-config") pod "73e5d206-2a90-4446-9eb5-cabfda46877f" (UID: "73e5d206-2a90-4446-9eb5-cabfda46877f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:12:56.652659 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652636 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "73e5d206-2a90-4446-9eb5-cabfda46877f" (UID: "73e5d206-2a90-4446-9eb5-cabfda46877f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:12:56.652857 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.652688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-service-ca" (OuterVolumeSpecName: "service-ca") pod "73e5d206-2a90-4446-9eb5-cabfda46877f" (UID: "73e5d206-2a90-4446-9eb5-cabfda46877f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:12:56.654498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.654468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73e5d206-2a90-4446-9eb5-cabfda46877f" (UID: "73e5d206-2a90-4446-9eb5-cabfda46877f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:12:56.654588 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.654537 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73e5d206-2a90-4446-9eb5-cabfda46877f" (UID: "73e5d206-2a90-4446-9eb5-cabfda46877f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:12:56.654588 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.654559 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e5d206-2a90-4446-9eb5-cabfda46877f-kube-api-access-5crw9" (OuterVolumeSpecName: "kube-api-access-5crw9") pod "73e5d206-2a90-4446-9eb5-cabfda46877f" (UID: "73e5d206-2a90-4446-9eb5-cabfda46877f"). InnerVolumeSpecName "kube-api-access-5crw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:12:56.753227 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.753193 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-trusted-ca-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:56.753227 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.753222 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5crw9\" (UniqueName: \"kubernetes.io/projected/73e5d206-2a90-4446-9eb5-cabfda46877f-kube-api-access-5crw9\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:56.753462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.753235 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-oauth-serving-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:56.753462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.753247 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:56.753462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.753259 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-oauth-config\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:56.753462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.753270 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e5d206-2a90-4446-9eb5-cabfda46877f-console-serving-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:56.753462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:56.753280 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e5d206-2a90-4446-9eb5-cabfda46877f-console-config\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:12:57.291462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.291424 2576 generic.go:356] "Generic (PLEG): container finished" podID="73e5d206-2a90-4446-9eb5-cabfda46877f" containerID="1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42" exitCode=0 Sep 10 19:12:57.291861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.291473 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bc79dbd-gdkhv" Sep 10 19:12:57.291861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.291486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bc79dbd-gdkhv" event={"ID":"73e5d206-2a90-4446-9eb5-cabfda46877f","Type":"ContainerDied","Data":"1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42"} Sep 10 19:12:57.291861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.291536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bc79dbd-gdkhv" event={"ID":"73e5d206-2a90-4446-9eb5-cabfda46877f","Type":"ContainerDied","Data":"75a75cc6442b8d32ac121a36c918dc070ab8c1f9fd0d7db5da3884829bdaa6b8"} Sep 10 19:12:57.291861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.291565 2576 scope.go:117] "RemoveContainer" containerID="1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42" Sep 10 19:12:57.300019 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.300004 2576 scope.go:117] "RemoveContainer" containerID="1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42" Sep 10 19:12:57.300295 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:12:57.300279 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42\": container with ID starting with 1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42 not found: ID does not exist" containerID="1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42" Sep 10 19:12:57.300336 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.300303 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42"} err="failed to get container status \"1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42\": rpc error: code = NotFound desc = could not find container \"1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42\": container with ID starting with 1e25641ea5088380f0fac741c9cd8deba9aa2ffae0c0c714d7508168332bcb42 not found: ID does not exist" Sep 10 19:12:57.311731 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.311709 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bc79dbd-gdkhv"] Sep 10 19:12:57.315470 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:57.315450 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55bc79dbd-gdkhv"] Sep 10 19:12:58.600739 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:12:58.600707 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e5d206-2a90-4446-9eb5-cabfda46877f" path="/var/lib/kubelet/pods/73e5d206-2a90-4446-9eb5-cabfda46877f/volumes" Sep 10 19:13:00.664856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:00.664824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:01.492078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:01.492047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:16.453643 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:16.453612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:16.570102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:16.570063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:18.352591 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.352556 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8678bcdcfb-n64x4"] Sep 10 19:13:18.353604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.353551 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e5d206-2a90-4446-9eb5-cabfda46877f" containerName="console" Sep 10 19:13:18.353604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.353582 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e5d206-2a90-4446-9eb5-cabfda46877f" containerName="console" Sep 10 19:13:18.353604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.353603 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b526d36-86c1-4af9-a501-7be9605f0e5d" containerName="registry" Sep 10 19:13:18.353604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.353611 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b526d36-86c1-4af9-a501-7be9605f0e5d" containerName="registry" Sep 10 19:13:18.353896 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.353701 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b526d36-86c1-4af9-a501-7be9605f0e5d" containerName="registry" Sep 10 19:13:18.353896 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.353715 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e5d206-2a90-4446-9eb5-cabfda46877f" containerName="console" Sep 10 19:13:18.358137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.358118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.366488 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.366463 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8678bcdcfb-n64x4"] Sep 10 19:13:18.416057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.416026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:18.481607 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.481579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:18.525724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.525687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-service-ca\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.525876 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.525735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82w5k\" (UniqueName: \"kubernetes.io/projected/aa1bd7eb-4971-4006-b359-dcac93f39ecd-kube-api-access-82w5k\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.525876 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.525828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-oauth-serving-cert\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.525876 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.525845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-trusted-ca-bundle\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.525876 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.525865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-serving-cert\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.526005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.525885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-config\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.526005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.525924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-oauth-config\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.626875 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.626786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-oauth-serving-cert\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.626875 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.626826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-trusted-ca-bundle\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.626875 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.626853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-serving-cert\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627148 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.626882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-config\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627148 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.626908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-oauth-config\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627148 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.626944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-service-ca\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627148 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.626979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82w5k\" (UniqueName: \"kubernetes.io/projected/aa1bd7eb-4971-4006-b359-dcac93f39ecd-kube-api-access-82w5k\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627734 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.627710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-trusted-ca-bundle\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.627711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-service-ca\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.627711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-config\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.627848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.627762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-oauth-serving-cert\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.629472 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.629442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-oauth-config\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.629551 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.629534 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-serving-cert\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.636511 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.636484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82w5k\" (UniqueName: \"kubernetes.io/projected/aa1bd7eb-4971-4006-b359-dcac93f39ecd-kube-api-access-82w5k\") pod \"console-8678bcdcfb-n64x4\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.668609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.668582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:18.946724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:18.946620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8678bcdcfb-n64x4"] Sep 10 19:13:18.951120 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:13:18.951090 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1bd7eb_4971_4006_b359_dcac93f39ecd.slice/crio-30fe4ea8da750ff7d1c2f8bf3e0d073de1478460760b68846ec58ce74e2d5e57 WatchSource:0}: Error finding container 30fe4ea8da750ff7d1c2f8bf3e0d073de1478460760b68846ec58ce74e2d5e57: Status 404 returned error can't find the container with id 30fe4ea8da750ff7d1c2f8bf3e0d073de1478460760b68846ec58ce74e2d5e57 Sep 10 19:13:19.378411 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:19.378362 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678bcdcfb-n64x4" event={"ID":"aa1bd7eb-4971-4006-b359-dcac93f39ecd","Type":"ContainerStarted","Data":"736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43"} Sep 10 19:13:19.378791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:19.378427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678bcdcfb-n64x4" event={"ID":"aa1bd7eb-4971-4006-b359-dcac93f39ecd","Type":"ContainerStarted","Data":"30fe4ea8da750ff7d1c2f8bf3e0d073de1478460760b68846ec58ce74e2d5e57"} Sep 10 19:13:19.397973 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:19.397923 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8678bcdcfb-n64x4" podStartSLOduration=1.397906661 podStartE2EDuration="1.397906661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:13:19.396196378 +0000 UTC m=+165.312852414" watchObservedRunningTime="2026-09-10 19:13:19.397906661 +0000 UTC m=+165.314562690" Sep 10 19:13:28.354557 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:28.354477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:28.455996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:28.455964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:28.669307 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:28.669222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:28.669307 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:28.669286 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:28.674111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:28.674088 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:29.419447 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:29.419421 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:13:29.469911 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:29.469883 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65f7df74b5-zx5bg"] Sep 10 19:13:29.603723 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:29.603690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:29.704926 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:29.704848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:29.776611 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:29.776581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:35.884200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:35.884167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:35.999629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:35.999600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:51.217356 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:51.217320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:13:54.491540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.491501 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65f7df74b5-zx5bg" podUID="5a057a7a-165a-4efd-9804-6ea1b5b289af" containerName="console" containerID="cri-o://883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7" gracePeriod=15 Sep 10 19:13:54.740101 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.740079 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:13:54.805292 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805265 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-service-ca\") pod \"5a057a7a-165a-4efd-9804-6ea1b5b289af\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " Sep 10 19:13:54.805485 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805298 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-trusted-ca-bundle\") pod \"5a057a7a-165a-4efd-9804-6ea1b5b289af\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " Sep 10 19:13:54.805485 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805334 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzwph\" (UniqueName: \"kubernetes.io/projected/5a057a7a-165a-4efd-9804-6ea1b5b289af-kube-api-access-nzwph\") pod \"5a057a7a-165a-4efd-9804-6ea1b5b289af\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " Sep 10 19:13:54.805485 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805366 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-oauth-config\") pod \"5a057a7a-165a-4efd-9804-6ea1b5b289af\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " Sep 10 19:13:54.805485 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805435 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-serving-cert\") pod \"5a057a7a-165a-4efd-9804-6ea1b5b289af\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " Sep 10 19:13:54.805485 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-config\") pod \"5a057a7a-165a-4efd-9804-6ea1b5b289af\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " Sep 10 19:13:54.805775 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805531 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-oauth-serving-cert\") pod \"5a057a7a-165a-4efd-9804-6ea1b5b289af\" (UID: \"5a057a7a-165a-4efd-9804-6ea1b5b289af\") " Sep 10 19:13:54.805878 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805848 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5a057a7a-165a-4efd-9804-6ea1b5b289af" (UID: "5a057a7a-165a-4efd-9804-6ea1b5b289af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:13:54.805946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805886 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-service-ca" (OuterVolumeSpecName: "service-ca") pod "5a057a7a-165a-4efd-9804-6ea1b5b289af" (UID: "5a057a7a-165a-4efd-9804-6ea1b5b289af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:13:54.805946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805893 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-config" (OuterVolumeSpecName: "console-config") pod "5a057a7a-165a-4efd-9804-6ea1b5b289af" (UID: "5a057a7a-165a-4efd-9804-6ea1b5b289af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:13:54.805946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.805921 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5a057a7a-165a-4efd-9804-6ea1b5b289af" (UID: "5a057a7a-165a-4efd-9804-6ea1b5b289af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:13:54.806111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.806046 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-config\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:13:54.806111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.806059 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-oauth-serving-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:13:54.806111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.806067 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:13:54.806111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.806075 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a057a7a-165a-4efd-9804-6ea1b5b289af-trusted-ca-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:13:54.807847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.807816 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5a057a7a-165a-4efd-9804-6ea1b5b289af" (UID: "5a057a7a-165a-4efd-9804-6ea1b5b289af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:13:54.807847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.807832 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5a057a7a-165a-4efd-9804-6ea1b5b289af" (UID: "5a057a7a-165a-4efd-9804-6ea1b5b289af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:13:54.807973 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.807901 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a057a7a-165a-4efd-9804-6ea1b5b289af-kube-api-access-nzwph" (OuterVolumeSpecName: "kube-api-access-nzwph") pod "5a057a7a-165a-4efd-9804-6ea1b5b289af" (UID: "5a057a7a-165a-4efd-9804-6ea1b5b289af"). InnerVolumeSpecName "kube-api-access-nzwph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:13:54.906744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.906715 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzwph\" (UniqueName: \"kubernetes.io/projected/5a057a7a-165a-4efd-9804-6ea1b5b289af-kube-api-access-nzwph\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:13:54.906744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.906742 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-oauth-config\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:13:54.906744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:54.906752 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a057a7a-165a-4efd-9804-6ea1b5b289af-console-serving-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:13:55.515305 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.515272 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a057a7a-165a-4efd-9804-6ea1b5b289af" containerID="883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7" exitCode=0 Sep 10 19:13:55.515711 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.515311 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f7df74b5-zx5bg" event={"ID":"5a057a7a-165a-4efd-9804-6ea1b5b289af","Type":"ContainerDied","Data":"883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7"} Sep 10 19:13:55.515711 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.515321 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f7df74b5-zx5bg" Sep 10 19:13:55.515711 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.515345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f7df74b5-zx5bg" event={"ID":"5a057a7a-165a-4efd-9804-6ea1b5b289af","Type":"ContainerDied","Data":"5051a3b431c6d16b2e89a0930b78905d1e33c30ccabca60a1bc71e6fe5b3b2ae"} Sep 10 19:13:55.515711 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.515363 2576 scope.go:117] "RemoveContainer" containerID="883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7" Sep 10 19:13:55.524351 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.524336 2576 scope.go:117] "RemoveContainer" containerID="883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7" Sep 10 19:13:55.524653 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:13:55.524634 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7\": container with ID starting with 883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7 not found: ID does not exist" containerID="883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7" Sep 10 19:13:55.524711 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.524663 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7"} err="failed to get container status \"883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7\": rpc error: code = NotFound desc = could not find container \"883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7\": container with ID starting with 883a22a96db5c80cd3c4f8eefcc88039c3210d65ddc524027aeb1c7ec7496ae7 not found: ID does not exist" Sep 10 19:13:55.536751 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.536725 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65f7df74b5-zx5bg"] Sep 10 19:13:55.539520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:55.539501 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65f7df74b5-zx5bg"] Sep 10 19:13:56.600473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:13:56.600440 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a057a7a-165a-4efd-9804-6ea1b5b289af" path="/var/lib/kubelet/pods/5a057a7a-165a-4efd-9804-6ea1b5b289af/volumes" Sep 10 19:14:01.140498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:01.140463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:04.851701 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:04.851666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:04.951322 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:04.951285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:05.617252 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.617220 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6"] Sep 10 19:14:05.617531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.617517 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a057a7a-165a-4efd-9804-6ea1b5b289af" containerName="console" Sep 10 19:14:05.617583 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.617532 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a057a7a-165a-4efd-9804-6ea1b5b289af" containerName="console" Sep 10 19:14:05.617623 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.617586 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a057a7a-165a-4efd-9804-6ea1b5b289af" containerName="console" Sep 10 19:14:05.622166 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.622150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.624869 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.624840 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 19:14:05.625910 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.625892 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ks6mr\"" Sep 10 19:14:05.626051 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.625900 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 19:14:05.629234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.629201 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6"] Sep 10 19:14:05.686918 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.686881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4dh\" (UniqueName: \"kubernetes.io/projected/23186e28-94f5-4cf0-b303-22de9c0946a9-kube-api-access-5p4dh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.687139 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.686924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.687139 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.686962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.787589 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.787558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4dh\" (UniqueName: \"kubernetes.io/projected/23186e28-94f5-4cf0-b303-22de9c0946a9-kube-api-access-5p4dh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.787700 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.787599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.787700 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.787634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.787927 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.787913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.787970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.787956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.796037 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.796016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4dh\" (UniqueName: \"kubernetes.io/projected/23186e28-94f5-4cf0-b303-22de9c0946a9-kube-api-access-5p4dh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:05.932317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:05.932233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:06.201279 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:06.201254 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6"] Sep 10 19:14:06.203818 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:14:06.203786 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23186e28_94f5_4cf0_b303_22de9c0946a9.slice/crio-d2e6c58679b43f11aa6e2d4b8b332a2cf02220e13c78bce9dc1801a0788e26b3 WatchSource:0}: Error finding container d2e6c58679b43f11aa6e2d4b8b332a2cf02220e13c78bce9dc1801a0788e26b3: Status 404 returned error can't find the container with id d2e6c58679b43f11aa6e2d4b8b332a2cf02220e13c78bce9dc1801a0788e26b3 Sep 10 19:14:06.558954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:06.558926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" event={"ID":"23186e28-94f5-4cf0-b303-22de9c0946a9","Type":"ContainerStarted","Data":"d2e6c58679b43f11aa6e2d4b8b332a2cf02220e13c78bce9dc1801a0788e26b3"} Sep 10 19:14:12.585507 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:12.585468 2576 generic.go:356] "Generic (PLEG): container finished" podID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerID="3b1e760924ddd0669249c5876aadca867eca6ed50ea6a80fca27ad38d9e3c7d6" exitCode=0 Sep 10 19:14:12.585888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:12.585558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" event={"ID":"23186e28-94f5-4cf0-b303-22de9c0946a9","Type":"ContainerDied","Data":"3b1e760924ddd0669249c5876aadca867eca6ed50ea6a80fca27ad38d9e3c7d6"} Sep 10 19:14:15.599227 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:15.599193 2576 generic.go:356] "Generic (PLEG): container finished" podID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerID="2fc7494449ef2632d6bbf8e5329ba3d8f0b6d16a7f7d46f5c41b3bd338a9208e" exitCode=0 Sep 10 19:14:15.599632 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:15.599280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" event={"ID":"23186e28-94f5-4cf0-b303-22de9c0946a9","Type":"ContainerDied","Data":"2fc7494449ef2632d6bbf8e5329ba3d8f0b6d16a7f7d46f5c41b3bd338a9208e"} Sep 10 19:14:22.627921 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:22.627882 2576 generic.go:356] "Generic (PLEG): container finished" podID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerID="a16a44e45e81d53dd36a390d369b7580db175353d01a979d2b65a54e8bca4dc3" exitCode=0 Sep 10 19:14:22.628328 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:22.627968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" event={"ID":"23186e28-94f5-4cf0-b303-22de9c0946a9","Type":"ContainerDied","Data":"a16a44e45e81d53dd36a390d369b7580db175353d01a979d2b65a54e8bca4dc3"} Sep 10 19:14:23.763013 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:23.762991 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:23.940681 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:23.940577 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-util\") pod \"23186e28-94f5-4cf0-b303-22de9c0946a9\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " Sep 10 19:14:23.940681 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:23.940632 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p4dh\" (UniqueName: \"kubernetes.io/projected/23186e28-94f5-4cf0-b303-22de9c0946a9-kube-api-access-5p4dh\") pod \"23186e28-94f5-4cf0-b303-22de9c0946a9\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " Sep 10 19:14:23.940940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:23.940731 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-bundle\") pod \"23186e28-94f5-4cf0-b303-22de9c0946a9\" (UID: \"23186e28-94f5-4cf0-b303-22de9c0946a9\") " Sep 10 19:14:23.941358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:23.941316 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-bundle" (OuterVolumeSpecName: "bundle") pod "23186e28-94f5-4cf0-b303-22de9c0946a9" (UID: "23186e28-94f5-4cf0-b303-22de9c0946a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:14:23.942987 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:23.942955 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23186e28-94f5-4cf0-b303-22de9c0946a9-kube-api-access-5p4dh" (OuterVolumeSpecName: "kube-api-access-5p4dh") pod "23186e28-94f5-4cf0-b303-22de9c0946a9" (UID: "23186e28-94f5-4cf0-b303-22de9c0946a9"). InnerVolumeSpecName "kube-api-access-5p4dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:14:23.944631 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:23.944610 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-util" (OuterVolumeSpecName: "util") pod "23186e28-94f5-4cf0-b303-22de9c0946a9" (UID: "23186e28-94f5-4cf0-b303-22de9c0946a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:14:24.042030 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:24.041984 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:14:24.042030 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:24.042020 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23186e28-94f5-4cf0-b303-22de9c0946a9-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:14:24.042030 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:24.042029 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4dh\" (UniqueName: \"kubernetes.io/projected/23186e28-94f5-4cf0-b303-22de9c0946a9-kube-api-access-5p4dh\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:14:24.636335 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:24.636299 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" event={"ID":"23186e28-94f5-4cf0-b303-22de9c0946a9","Type":"ContainerDied","Data":"d2e6c58679b43f11aa6e2d4b8b332a2cf02220e13c78bce9dc1801a0788e26b3"} Sep 10 19:14:24.636500 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:24.636344 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e6c58679b43f11aa6e2d4b8b332a2cf02220e13c78bce9dc1801a0788e26b3" Sep 10 19:14:24.636500 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:24.636313 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lclg6" Sep 10 19:14:25.098996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:25.098961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:26.717674 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:26.717639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:27.850293 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:27.850259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:29.321049 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321016 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8"] Sep 10 19:14:29.321430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321279 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerName="util" Sep 10 19:14:29.321430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321289 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerName="util" Sep 10 19:14:29.321430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321295 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerName="pull" Sep 10 19:14:29.321430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321300 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerName="pull" Sep 10 19:14:29.321430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321309 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerName="extract" Sep 10 19:14:29.321430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321315 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerName="extract" Sep 10 19:14:29.321430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.321394 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23186e28-94f5-4cf0-b303-22de9c0946a9" containerName="extract" Sep 10 19:14:29.324211 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.324193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.327013 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.326992 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 19:14:29.327254 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.327227 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5dm8c\"" Sep 10 19:14:29.327409 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.327392 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:14:29.342154 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.342131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8"] Sep 10 19:14:29.490717 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.490680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca478476-edcf-45e5-8b15-60214aa24a40-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6h4m8\" (UID: \"ca478476-edcf-45e5-8b15-60214aa24a40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.490717 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.490717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpdj\" (UniqueName: \"kubernetes.io/projected/ca478476-edcf-45e5-8b15-60214aa24a40-kube-api-access-xmpdj\") pod \"cert-manager-operator-controller-manager-55c444bf86-6h4m8\" (UID: \"ca478476-edcf-45e5-8b15-60214aa24a40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.591310 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.591232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca478476-edcf-45e5-8b15-60214aa24a40-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6h4m8\" (UID: \"ca478476-edcf-45e5-8b15-60214aa24a40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.591310 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.591262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpdj\" (UniqueName: \"kubernetes.io/projected/ca478476-edcf-45e5-8b15-60214aa24a40-kube-api-access-xmpdj\") pod \"cert-manager-operator-controller-manager-55c444bf86-6h4m8\" (UID: \"ca478476-edcf-45e5-8b15-60214aa24a40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.591686 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.591665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca478476-edcf-45e5-8b15-60214aa24a40-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6h4m8\" (UID: \"ca478476-edcf-45e5-8b15-60214aa24a40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.601382 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.601348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpdj\" (UniqueName: \"kubernetes.io/projected/ca478476-edcf-45e5-8b15-60214aa24a40-kube-api-access-xmpdj\") pod \"cert-manager-operator-controller-manager-55c444bf86-6h4m8\" (UID: \"ca478476-edcf-45e5-8b15-60214aa24a40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.633578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.633551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" Sep 10 19:14:29.912575 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:29.912550 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8"] Sep 10 19:14:29.915327 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:14:29.915290 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca478476_edcf_45e5_8b15_60214aa24a40.slice/crio-2f1b401464960908b60898a1c7de43aa9a5a69b87c3540dbf1537cec8949d465 WatchSource:0}: Error finding container 2f1b401464960908b60898a1c7de43aa9a5a69b87c3540dbf1537cec8949d465: Status 404 returned error can't find the container with id 2f1b401464960908b60898a1c7de43aa9a5a69b87c3540dbf1537cec8949d465 Sep 10 19:14:30.659686 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:30.659655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" event={"ID":"ca478476-edcf-45e5-8b15-60214aa24a40","Type":"ContainerStarted","Data":"2f1b401464960908b60898a1c7de43aa9a5a69b87c3540dbf1537cec8949d465"} Sep 10 19:14:30.767944 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:30.767908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:30.871600 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:30.871569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:32.448546 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:32.448509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:32.589646 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:32.589607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:32.670137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:32.670107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" event={"ID":"ca478476-edcf-45e5-8b15-60214aa24a40","Type":"ContainerStarted","Data":"453ce08edba61a18a9a1f3893732c2315892f9683f32916247d0cc8edd7112e0"} Sep 10 19:14:32.691993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:32.691943 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6h4m8" podStartSLOduration=1.988632361 podStartE2EDuration="3.691927327s" totalImagesPullingTime="1.703294966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:14:32.691204457 +0000 UTC m=+238.607860501" watchObservedRunningTime="2026-09-10 19:14:32.691927327 +0000 UTC m=+238.608583356" Sep 10 19:14:33.198246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.198211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:33.400492 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.400460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:33.507927 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.507848 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d"] Sep 10 19:14:33.511113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.511090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.513749 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.513726 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 19:14:33.514060 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.514031 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 19:14:33.514926 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.514912 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ks6mr\"" Sep 10 19:14:33.521019 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.520998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.521107 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.521031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqckr\" (UniqueName: \"kubernetes.io/projected/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-kube-api-access-bqckr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.521107 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.521052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.529616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.529593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d"] Sep 10 19:14:33.621532 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.621499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.621718 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.621543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqckr\" (UniqueName: \"kubernetes.io/projected/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-kube-api-access-bqckr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.621718 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.621565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.621947 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.621923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.622012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.621973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.635447 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.635417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqckr\" (UniqueName: \"kubernetes.io/projected/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-kube-api-access-bqckr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:33.820469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:33.820426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:34.050634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.050606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:34.094963 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.094889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d"] Sep 10 19:14:34.097939 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:14:34.097915 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a08ba9e_06cd_490d_b3c9_698cc5dc071e.slice/crio-07a8174494120c7de17288d2aaad5568cb63270642ad56784fd82965cc30102a WatchSource:0}: Error finding container 07a8174494120c7de17288d2aaad5568cb63270642ad56784fd82965cc30102a: Status 404 returned error can't find the container with id 07a8174494120c7de17288d2aaad5568cb63270642ad56784fd82965cc30102a Sep 10 19:14:34.191686 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.191647 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h"] Sep 10 19:14:34.194868 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.194851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.197709 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.197686 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 19:14:34.197819 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.197704 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 19:14:34.197819 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.197798 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nn9cx\"" Sep 10 19:14:34.208200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.208179 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h"] Sep 10 19:14:34.225413 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.225363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9vh\" (UniqueName: \"kubernetes.io/projected/57692a4c-c53c-46ab-b519-6cfcd8bdf607-kube-api-access-lf9vh\") pod \"cert-manager-cainjector-559cdd8c87-6xc5h\" (UID: \"57692a4c-c53c-46ab-b519-6cfcd8bdf607\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.225532 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.225418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57692a4c-c53c-46ab-b519-6cfcd8bdf607-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6xc5h\" (UID: \"57692a4c-c53c-46ab-b519-6cfcd8bdf607\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.326418 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.326360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9vh\" (UniqueName: \"kubernetes.io/projected/57692a4c-c53c-46ab-b519-6cfcd8bdf607-kube-api-access-lf9vh\") pod \"cert-manager-cainjector-559cdd8c87-6xc5h\" (UID: \"57692a4c-c53c-46ab-b519-6cfcd8bdf607\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.326418 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.326422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57692a4c-c53c-46ab-b519-6cfcd8bdf607-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6xc5h\" (UID: \"57692a4c-c53c-46ab-b519-6cfcd8bdf607\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.335421 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.335390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57692a4c-c53c-46ab-b519-6cfcd8bdf607-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6xc5h\" (UID: \"57692a4c-c53c-46ab-b519-6cfcd8bdf607\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.335527 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.335432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9vh\" (UniqueName: \"kubernetes.io/projected/57692a4c-c53c-46ab-b519-6cfcd8bdf607-kube-api-access-lf9vh\") pod \"cert-manager-cainjector-559cdd8c87-6xc5h\" (UID: \"57692a4c-c53c-46ab-b519-6cfcd8bdf607\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.579762 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.579732 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nn9cx\"" Sep 10 19:14:34.587830 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.587804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" Sep 10 19:14:34.680102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.680066 2576 generic.go:356] "Generic (PLEG): container finished" podID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerID="8accfbcc29e2f0dd86a60d1ba26726e176a9d5a50b1769ca96d6ca20229aa6f2" exitCode=0 Sep 10 19:14:34.680256 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.680159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" event={"ID":"0a08ba9e-06cd-490d-b3c9-698cc5dc071e","Type":"ContainerDied","Data":"8accfbcc29e2f0dd86a60d1ba26726e176a9d5a50b1769ca96d6ca20229aa6f2"} Sep 10 19:14:34.680256 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.680210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" event={"ID":"0a08ba9e-06cd-490d-b3c9-698cc5dc071e","Type":"ContainerStarted","Data":"07a8174494120c7de17288d2aaad5568cb63270642ad56784fd82965cc30102a"} Sep 10 19:14:34.866654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.866443 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h"] Sep 10 19:14:34.869007 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:14:34.868983 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57692a4c_c53c_46ab_b519_6cfcd8bdf607.slice/crio-5edd5a414dc085b15375a25b7f61c3454a13a29292beb2f294a0d03148497efd WatchSource:0}: Error finding container 5edd5a414dc085b15375a25b7f61c3454a13a29292beb2f294a0d03148497efd: Status 404 returned error can't find the container with id 5edd5a414dc085b15375a25b7f61c3454a13a29292beb2f294a0d03148497efd Sep 10 19:14:34.883686 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:34.883653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:35.687130 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:35.687085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" event={"ID":"57692a4c-c53c-46ab-b519-6cfcd8bdf607","Type":"ContainerStarted","Data":"5edd5a414dc085b15375a25b7f61c3454a13a29292beb2f294a0d03148497efd"} Sep 10 19:14:37.607479 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:37.607448 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:37.689737 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:37.689703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:38.702785 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:38.702750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" event={"ID":"57692a4c-c53c-46ab-b519-6cfcd8bdf607","Type":"ContainerStarted","Data":"9374c22646e31127c722818a23041308e13ae426c761ae1d2d3e98e395495278"} Sep 10 19:14:38.704568 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:38.704544 2576 generic.go:356] "Generic (PLEG): container finished" podID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerID="3bde66cbe61e3e13ec137b7328cfd8d4bdf9ab4d20f0bc16535fa94817d58592" exitCode=0 Sep 10 19:14:38.704675 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:38.704621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" event={"ID":"0a08ba9e-06cd-490d-b3c9-698cc5dc071e","Type":"ContainerDied","Data":"3bde66cbe61e3e13ec137b7328cfd8d4bdf9ab4d20f0bc16535fa94817d58592"} Sep 10 19:14:38.724861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:38.724818 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xc5h" podStartSLOduration=1.676962932 podStartE2EDuration="4.724801017s" totalImagesPullingTime="3.047838085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:14:38.723034083 +0000 UTC m=+244.639690114" watchObservedRunningTime="2026-09-10 19:14:38.724801017 +0000 UTC m=+244.641457037" Sep 10 19:14:39.710783 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:39.710749 2576 generic.go:356] "Generic (PLEG): container finished" podID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerID="a167fe1fab44ed808818c7e30796956d613a8c423d38e1d1fa78d7ba0ed4f774" exitCode=0 Sep 10 19:14:39.711183 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:39.710837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" event={"ID":"0a08ba9e-06cd-490d-b3c9-698cc5dc071e","Type":"ContainerDied","Data":"a167fe1fab44ed808818c7e30796956d613a8c423d38e1d1fa78d7ba0ed4f774"} Sep 10 19:14:40.838936 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.838910 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:40.876931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.876836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-util\") pod \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " Sep 10 19:14:40.877062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.876983 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqckr\" (UniqueName: \"kubernetes.io/projected/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-kube-api-access-bqckr\") pod \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " Sep 10 19:14:40.877062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.877022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-bundle\") pod \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\" (UID: \"0a08ba9e-06cd-490d-b3c9-698cc5dc071e\") " Sep 10 19:14:40.877474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.877447 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-bundle" (OuterVolumeSpecName: "bundle") pod "0a08ba9e-06cd-490d-b3c9-698cc5dc071e" (UID: "0a08ba9e-06cd-490d-b3c9-698cc5dc071e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:14:40.879235 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.879202 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-kube-api-access-bqckr" (OuterVolumeSpecName: "kube-api-access-bqckr") pod "0a08ba9e-06cd-490d-b3c9-698cc5dc071e" (UID: "0a08ba9e-06cd-490d-b3c9-698cc5dc071e"). InnerVolumeSpecName "kube-api-access-bqckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:14:40.880996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.880976 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-util" (OuterVolumeSpecName: "util") pod "0a08ba9e-06cd-490d-b3c9-698cc5dc071e" (UID: "0a08ba9e-06cd-490d-b3c9-698cc5dc071e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:14:40.977744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.977657 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqckr\" (UniqueName: \"kubernetes.io/projected/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-kube-api-access-bqckr\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:14:40.977744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.977683 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:14:40.977744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:40.977694 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a08ba9e-06cd-490d-b3c9-698cc5dc071e-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:14:41.720846 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:41.720815 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" Sep 10 19:14:41.720846 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:41.720827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsgj2d" event={"ID":"0a08ba9e-06cd-490d-b3c9-698cc5dc071e","Type":"ContainerDied","Data":"07a8174494120c7de17288d2aaad5568cb63270642ad56784fd82965cc30102a"} Sep 10 19:14:41.721043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:41.720869 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a8174494120c7de17288d2aaad5568cb63270642ad56784fd82965cc30102a" Sep 10 19:14:43.733925 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:43.733888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:44.305615 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:44.305585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:44.474450 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:44.474416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:49.420477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:49.420447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:49.579320 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:49.579287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:51.369231 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369197 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r"] Sep 10 19:14:51.369622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369514 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerName="pull" Sep 10 19:14:51.369622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369529 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerName="pull" Sep 10 19:14:51.369622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369539 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerName="extract" Sep 10 19:14:51.369622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369544 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerName="extract" Sep 10 19:14:51.369622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369552 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerName="util" Sep 10 19:14:51.369622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369558 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerName="util" Sep 10 19:14:51.369622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.369617 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a08ba9e-06cd-490d-b3c9-698cc5dc071e" containerName="extract" Sep 10 19:14:51.376743 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.376717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.379290 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.379260 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:14:51.379424 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.379346 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 19:14:51.380264 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.380248 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6494w\"" Sep 10 19:14:51.385955 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.385932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r"] Sep 10 19:14:51.452534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.452499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5hq4r\" (UID: \"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.452701 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.452600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8bc\" (UniqueName: \"kubernetes.io/projected/3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47-kube-api-access-6t8bc\") pod \"openshift-lws-operator-fd8ccff4c-5hq4r\" (UID: \"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.554000 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.553964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8bc\" (UniqueName: \"kubernetes.io/projected/3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47-kube-api-access-6t8bc\") pod \"openshift-lws-operator-fd8ccff4c-5hq4r\" (UID: \"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.554173 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.554018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5hq4r\" (UID: \"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.554351 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.554336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5hq4r\" (UID: \"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.568759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.568729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8bc\" (UniqueName: \"kubernetes.io/projected/3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47-kube-api-access-6t8bc\") pod \"openshift-lws-operator-fd8ccff4c-5hq4r\" (UID: \"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.685821 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.685738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" Sep 10 19:14:51.959083 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:51.959060 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r"] Sep 10 19:14:51.961580 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:14:51.961552 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fdb8b2a_1ac1_4a02_b9cb_eb26955fbb47.slice/crio-407d2a1e3de80671708c6bfd88dbf4b2d3948a1831b77dfb934e7e2eb5ea18fd WatchSource:0}: Error finding container 407d2a1e3de80671708c6bfd88dbf4b2d3948a1831b77dfb934e7e2eb5ea18fd: Status 404 returned error can't find the container with id 407d2a1e3de80671708c6bfd88dbf4b2d3948a1831b77dfb934e7e2eb5ea18fd Sep 10 19:14:52.772258 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:52.772197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" event={"ID":"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47","Type":"ContainerStarted","Data":"407d2a1e3de80671708c6bfd88dbf4b2d3948a1831b77dfb934e7e2eb5ea18fd"} Sep 10 19:14:54.785524 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:54.785489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" event={"ID":"3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47","Type":"ContainerStarted","Data":"045e76e7866efc4717245705ef744257775803baf20de026ac542cc9ff7e91b2"} Sep 10 19:14:54.808773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:54.808723 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5hq4r" podStartSLOduration=2.0355360239999998 podStartE2EDuration="3.808701911s" totalImagesPullingTime="1.773165887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:14:54.807903347 +0000 UTC m=+260.724559378" watchObservedRunningTime="2026-09-10 19:14:54.808701911 +0000 UTC m=+260.725357940" Sep 10 19:14:55.482627 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:55.482589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:55.582531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:55.582493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:56.981725 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:56.981648 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56"] Sep 10 19:14:57.006653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.006629 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56"] Sep 10 19:14:57.006814 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.006721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.009991 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.009969 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ks6mr\"" Sep 10 19:14:57.010297 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.010281 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 19:14:57.010948 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.010931 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 19:14:57.102365 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.102327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq2pr\" (UniqueName: \"kubernetes.io/projected/de2bb9e7-2aaf-4750-9a48-762ad68dc031-kube-api-access-kq2pr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.102566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.102397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.102566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.102453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.158405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.158354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:14:57.203653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.203616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.203809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.203684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq2pr\" (UniqueName: \"kubernetes.io/projected/de2bb9e7-2aaf-4750-9a48-762ad68dc031-kube-api-access-kq2pr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.203809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.203713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.204009 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.203989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.204096 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.204010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.216867 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.216841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq2pr\" (UniqueName: \"kubernetes.io/projected/de2bb9e7-2aaf-4750-9a48-762ad68dc031-kube-api-access-kq2pr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.315952 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.315924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:14:57.621667 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.621641 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56"] Sep 10 19:14:57.624403 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:14:57.624359 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2bb9e7_2aaf_4750_9a48_762ad68dc031.slice/crio-5b219d99fbc31d72380e81bb29789f7f567ec2f86bd5f2c4fbc0d8bc2991d221 WatchSource:0}: Error finding container 5b219d99fbc31d72380e81bb29789f7f567ec2f86bd5f2c4fbc0d8bc2991d221: Status 404 returned error can't find the container with id 5b219d99fbc31d72380e81bb29789f7f567ec2f86bd5f2c4fbc0d8bc2991d221 Sep 10 19:14:57.799505 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.799477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" event={"ID":"de2bb9e7-2aaf-4750-9a48-762ad68dc031","Type":"ContainerStarted","Data":"e8cf263f38cd80b338f96420719e41e4c035d50b69688df434cf3d345545d7c7"} Sep 10 19:14:57.799636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:57.799522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" event={"ID":"de2bb9e7-2aaf-4750-9a48-762ad68dc031","Type":"ContainerStarted","Data":"5b219d99fbc31d72380e81bb29789f7f567ec2f86bd5f2c4fbc0d8bc2991d221"} Sep 10 19:14:58.804890 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:58.804855 2576 generic.go:356] "Generic (PLEG): container finished" podID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerID="e8cf263f38cd80b338f96420719e41e4c035d50b69688df434cf3d345545d7c7" exitCode=0 Sep 10 19:14:58.805257 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:58.804948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" event={"ID":"de2bb9e7-2aaf-4750-9a48-762ad68dc031","Type":"ContainerDied","Data":"e8cf263f38cd80b338f96420719e41e4c035d50b69688df434cf3d345545d7c7"} Sep 10 19:14:59.811000 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:59.810967 2576 generic.go:356] "Generic (PLEG): container finished" podID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerID="5d42c4eded37db015b4403875d3b62b7a091d250940b1fc83efd320254b939d3" exitCode=0 Sep 10 19:14:59.811464 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:14:59.811057 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" event={"ID":"de2bb9e7-2aaf-4750-9a48-762ad68dc031","Type":"ContainerDied","Data":"5d42c4eded37db015b4403875d3b62b7a091d250940b1fc83efd320254b939d3"} Sep 10 19:15:00.817747 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:00.817717 2576 generic.go:356] "Generic (PLEG): container finished" podID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerID="6dd4218ed7aa2c8c9a9109480478954859f1ef4659a63b8fde37a5f23b094dd5" exitCode=0 Sep 10 19:15:00.818141 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:00.817804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" event={"ID":"de2bb9e7-2aaf-4750-9a48-762ad68dc031","Type":"ContainerDied","Data":"6dd4218ed7aa2c8c9a9109480478954859f1ef4659a63b8fde37a5f23b094dd5"} Sep 10 19:15:01.950501 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:01.950469 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:15:02.045588 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.045555 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq2pr\" (UniqueName: \"kubernetes.io/projected/de2bb9e7-2aaf-4750-9a48-762ad68dc031-kube-api-access-kq2pr\") pod \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " Sep 10 19:15:02.045781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.045604 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-util\") pod \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " Sep 10 19:15:02.045781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.045664 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-bundle\") pod \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\" (UID: \"de2bb9e7-2aaf-4750-9a48-762ad68dc031\") " Sep 10 19:15:02.046367 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.046340 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-bundle" (OuterVolumeSpecName: "bundle") pod "de2bb9e7-2aaf-4750-9a48-762ad68dc031" (UID: "de2bb9e7-2aaf-4750-9a48-762ad68dc031"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:15:02.047853 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.047828 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2bb9e7-2aaf-4750-9a48-762ad68dc031-kube-api-access-kq2pr" (OuterVolumeSpecName: "kube-api-access-kq2pr") pod "de2bb9e7-2aaf-4750-9a48-762ad68dc031" (UID: "de2bb9e7-2aaf-4750-9a48-762ad68dc031"). InnerVolumeSpecName "kube-api-access-kq2pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:15:02.051123 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.051085 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-util" (OuterVolumeSpecName: "util") pod "de2bb9e7-2aaf-4750-9a48-762ad68dc031" (UID: "de2bb9e7-2aaf-4750-9a48-762ad68dc031"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:15:02.146625 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.146545 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:15:02.146625 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.146573 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de2bb9e7-2aaf-4750-9a48-762ad68dc031-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:15:02.146625 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.146582 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq2pr\" (UniqueName: \"kubernetes.io/projected/de2bb9e7-2aaf-4750-9a48-762ad68dc031-kube-api-access-kq2pr\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:15:02.827969 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.827940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" event={"ID":"de2bb9e7-2aaf-4750-9a48-762ad68dc031","Type":"ContainerDied","Data":"5b219d99fbc31d72380e81bb29789f7f567ec2f86bd5f2c4fbc0d8bc2991d221"} Sep 10 19:15:02.827969 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.827964 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7t56" Sep 10 19:15:02.828171 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:02.827985 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b219d99fbc31d72380e81bb29789f7f567ec2f86bd5f2c4fbc0d8bc2991d221" Sep 10 19:15:07.447873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.447842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:07.539478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539442 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz"] Sep 10 19:15:07.539759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539746 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerName="util" Sep 10 19:15:07.539799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539761 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerName="util" Sep 10 19:15:07.539799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539769 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerName="extract" Sep 10 19:15:07.539799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539775 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerName="extract" Sep 10 19:15:07.539799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539790 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerName="pull" Sep 10 19:15:07.539799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539799 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerName="pull" Sep 10 19:15:07.539947 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.539856 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2bb9e7-2aaf-4750-9a48-762ad68dc031" containerName="extract" Sep 10 19:15:07.545115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.545097 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.548363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.548336 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 19:15:07.549305 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.549285 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 19:15:07.553158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.553137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 19:15:07.554061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.554045 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jlvx4\"" Sep 10 19:15:07.584216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.584190 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz"] Sep 10 19:15:07.620647 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.620618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:07.691099 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.691064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09d027f9-967d-45bc-8770-303097899376-manager-config\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.691257 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.691123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09d027f9-967d-45bc-8770-303097899376-metrics-cert\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.691257 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.691178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d027f9-967d-45bc-8770-303097899376-cert\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.691257 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.691212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7pr\" (UniqueName: \"kubernetes.io/projected/09d027f9-967d-45bc-8770-303097899376-kube-api-access-2m7pr\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.792141 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.792108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d027f9-967d-45bc-8770-303097899376-cert\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.792289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.792150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7pr\" (UniqueName: \"kubernetes.io/projected/09d027f9-967d-45bc-8770-303097899376-kube-api-access-2m7pr\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.792289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.792203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09d027f9-967d-45bc-8770-303097899376-manager-config\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.792289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.792249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09d027f9-967d-45bc-8770-303097899376-metrics-cert\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.793058 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.792982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09d027f9-967d-45bc-8770-303097899376-manager-config\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.794945 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.794916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d027f9-967d-45bc-8770-303097899376-cert\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.795055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.794960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09d027f9-967d-45bc-8770-303097899376-metrics-cert\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.806186 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.806163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7pr\" (UniqueName: \"kubernetes.io/projected/09d027f9-967d-45bc-8770-303097899376-kube-api-access-2m7pr\") pod \"lws-controller-manager-6748c6bd4b-5q9jz\" (UID: \"09d027f9-967d-45bc-8770-303097899376\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.812873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.812851 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk"] Sep 10 19:15:07.817109 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.817094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.819679 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.819661 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 19:15:07.819776 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.819709 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 19:15:07.819776 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.819711 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ks6mr\"" Sep 10 19:15:07.828299 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.828280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk"] Sep 10 19:15:07.854426 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.854404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:07.892840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.892800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.893018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.892863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.893018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.892937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwttk\" (UniqueName: \"kubernetes.io/projected/6821e606-b846-4c98-a7c9-ecde0395368c-kube-api-access-zwttk\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.994352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.994307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwttk\" (UniqueName: \"kubernetes.io/projected/6821e606-b846-4c98-a7c9-ecde0395368c-kube-api-access-zwttk\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.994565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.994367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.994565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.994419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.994845 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.994824 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:07.994901 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:07.994841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:08.011007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.010978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwttk\" (UniqueName: \"kubernetes.io/projected/6821e606-b846-4c98-a7c9-ecde0395368c-kube-api-access-zwttk\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:08.126129 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.126064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:08.161896 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.161865 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz"] Sep 10 19:15:08.165347 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:15:08.165316 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d027f9_967d_45bc_8770_303097899376.slice/crio-109e1282f14f91afe5375126c9b3d5407ba5db97119e4bb3751ae2a9845e1ad5 WatchSource:0}: Error finding container 109e1282f14f91afe5375126c9b3d5407ba5db97119e4bb3751ae2a9845e1ad5: Status 404 returned error can't find the container with id 109e1282f14f91afe5375126c9b3d5407ba5db97119e4bb3751ae2a9845e1ad5 Sep 10 19:15:08.408113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.408079 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk"] Sep 10 19:15:08.410246 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:15:08.410207 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6821e606_b846_4c98_a7c9_ecde0395368c.slice/crio-a4efb346b313c0f9c6941a38f5dfdb3a09fb82aed8411afa9fc0baa6bb3cbd75 WatchSource:0}: Error finding container a4efb346b313c0f9c6941a38f5dfdb3a09fb82aed8411afa9fc0baa6bb3cbd75: Status 404 returned error can't find the container with id a4efb346b313c0f9c6941a38f5dfdb3a09fb82aed8411afa9fc0baa6bb3cbd75 Sep 10 19:15:08.854929 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.854895 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" event={"ID":"09d027f9-967d-45bc-8770-303097899376","Type":"ContainerStarted","Data":"109e1282f14f91afe5375126c9b3d5407ba5db97119e4bb3751ae2a9845e1ad5"} Sep 10 19:15:08.856640 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.856612 2576 generic.go:356] "Generic (PLEG): container finished" podID="6821e606-b846-4c98-a7c9-ecde0395368c" containerID="53e2ade856d84c6714e166283066cb01cd146e80d5853bee49b702791de1eb8e" exitCode=0 Sep 10 19:15:08.856767 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.856661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" event={"ID":"6821e606-b846-4c98-a7c9-ecde0395368c","Type":"ContainerDied","Data":"53e2ade856d84c6714e166283066cb01cd146e80d5853bee49b702791de1eb8e"} Sep 10 19:15:08.856767 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.856700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" event={"ID":"6821e606-b846-4c98-a7c9-ecde0395368c","Type":"ContainerStarted","Data":"a4efb346b313c0f9c6941a38f5dfdb3a09fb82aed8411afa9fc0baa6bb3cbd75"} Sep 10 19:15:08.975563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:08.975530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:09.132082 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:09.131997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:09.674222 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:09.674173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:10.135248 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.135185 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:10.253509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.253462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:10.362417 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.362363 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x"] Sep 10 19:15:10.372082 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.372061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.376569 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.376546 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 19:15:10.376908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.376889 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-574n2\"" Sep 10 19:15:10.377102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.377087 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 19:15:10.377713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.377696 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 19:15:10.382156 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.382141 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 19:15:10.402266 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.402199 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x"] Sep 10 19:15:10.515297 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.515263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c03473-d443-4ee3-b9ad-18d41c71da27-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.515521 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.515319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2c03473-d443-4ee3-b9ad-18d41c71da27-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.515521 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.515411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2c03473-d443-4ee3-b9ad-18d41c71da27-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.515609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.515521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4kd\" (UniqueName: \"kubernetes.io/projected/c2c03473-d443-4ee3-b9ad-18d41c71da27-kube-api-access-jf4kd\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.515609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.515576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2c03473-d443-4ee3-b9ad-18d41c71da27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.535929 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.535906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:10.616943 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.616914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c03473-d443-4ee3-b9ad-18d41c71da27-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.617101 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.616964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2c03473-d443-4ee3-b9ad-18d41c71da27-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.617167 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.617110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2c03473-d443-4ee3-b9ad-18d41c71da27-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.617248 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.617214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4kd\" (UniqueName: \"kubernetes.io/projected/c2c03473-d443-4ee3-b9ad-18d41c71da27-kube-api-access-jf4kd\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.617307 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.617274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2c03473-d443-4ee3-b9ad-18d41c71da27-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.617396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.617301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2c03473-d443-4ee3-b9ad-18d41c71da27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.619809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.619784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c03473-d443-4ee3-b9ad-18d41c71da27-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.620056 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.620037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2c03473-d443-4ee3-b9ad-18d41c71da27-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.620105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.620043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2c03473-d443-4ee3-b9ad-18d41c71da27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.631203 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.631182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4kd\" (UniqueName: \"kubernetes.io/projected/c2c03473-d443-4ee3-b9ad-18d41c71da27-kube-api-access-jf4kd\") pod \"opendatahub-operator-controller-manager-d54bdf466-gct5x\" (UID: \"c2c03473-d443-4ee3-b9ad-18d41c71da27\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.653983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.653907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:10.711768 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.711731 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:10.787057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.787026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:10.868084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.868041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" event={"ID":"09d027f9-967d-45bc-8770-303097899376","Type":"ContainerStarted","Data":"387d9ae946a81f2de80f9ec8b3f6ee57a4d01c55102c8291ea3fa1b8d74207b6"} Sep 10 19:15:10.868261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.868117 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:10.869970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.869943 2576 generic.go:356] "Generic (PLEG): container finished" podID="6821e606-b846-4c98-a7c9-ecde0395368c" containerID="15d6b1137801d6bbf9bf6222dfe762727dce14d47888bf19bb32d9ddcb18f058" exitCode=0 Sep 10 19:15:10.870104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.869987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" event={"ID":"6821e606-b846-4c98-a7c9-ecde0395368c","Type":"ContainerDied","Data":"15d6b1137801d6bbf9bf6222dfe762727dce14d47888bf19bb32d9ddcb18f058"} Sep 10 19:15:10.892854 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.892805 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" podStartSLOduration=1.7121456290000001 podStartE2EDuration="3.892788676s" totalImagesPullingTime="2.180643047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:10.89145282 +0000 UTC m=+276.808108850" watchObservedRunningTime="2026-09-10 19:15:10.892788676 +0000 UTC m=+276.809444704" Sep 10 19:15:10.913781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:10.913756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:11.002198 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:11.002176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x"] Sep 10 19:15:11.004257 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:15:11.004228 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c03473_d443_4ee3_b9ad_18d41c71da27.slice/crio-d5180b617c3a28e396c8793f27569981ae41a612be1d348180a2224623f631c6 WatchSource:0}: Error finding container d5180b617c3a28e396c8793f27569981ae41a612be1d348180a2224623f631c6: Status 404 returned error can't find the container with id d5180b617c3a28e396c8793f27569981ae41a612be1d348180a2224623f631c6 Sep 10 19:15:11.525279 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:11.525247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:11.624515 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:11.624434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:11.879279 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:11.879185 2576 generic.go:356] "Generic (PLEG): container finished" podID="6821e606-b846-4c98-a7c9-ecde0395368c" containerID="5d30ae012300d83912cef9cfebba27f57b53ccea2eddca0e86625b437aeeb688" exitCode=0 Sep 10 19:15:11.879279 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:11.879269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" event={"ID":"6821e606-b846-4c98-a7c9-ecde0395368c","Type":"ContainerDied","Data":"5d30ae012300d83912cef9cfebba27f57b53ccea2eddca0e86625b437aeeb688"} Sep 10 19:15:11.881222 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:11.881193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" event={"ID":"c2c03473-d443-4ee3-b9ad-18d41c71da27","Type":"ContainerStarted","Data":"d5180b617c3a28e396c8793f27569981ae41a612be1d348180a2224623f631c6"} Sep 10 19:15:14.182124 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.182100 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:14.246486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.246454 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-bundle\") pod \"6821e606-b846-4c98-a7c9-ecde0395368c\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " Sep 10 19:15:14.246638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.246544 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-util\") pod \"6821e606-b846-4c98-a7c9-ecde0395368c\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " Sep 10 19:15:14.246638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.246633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwttk\" (UniqueName: \"kubernetes.io/projected/6821e606-b846-4c98-a7c9-ecde0395368c-kube-api-access-zwttk\") pod \"6821e606-b846-4c98-a7c9-ecde0395368c\" (UID: \"6821e606-b846-4c98-a7c9-ecde0395368c\") " Sep 10 19:15:14.247417 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.247389 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-bundle" (OuterVolumeSpecName: "bundle") pod "6821e606-b846-4c98-a7c9-ecde0395368c" (UID: "6821e606-b846-4c98-a7c9-ecde0395368c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:15:14.248932 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.248909 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6821e606-b846-4c98-a7c9-ecde0395368c-kube-api-access-zwttk" (OuterVolumeSpecName: "kube-api-access-zwttk") pod "6821e606-b846-4c98-a7c9-ecde0395368c" (UID: "6821e606-b846-4c98-a7c9-ecde0395368c"). InnerVolumeSpecName "kube-api-access-zwttk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:15:14.252067 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.252046 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-util" (OuterVolumeSpecName: "util") pod "6821e606-b846-4c98-a7c9-ecde0395368c" (UID: "6821e606-b846-4c98-a7c9-ecde0395368c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:15:14.347486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.347453 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwttk\" (UniqueName: \"kubernetes.io/projected/6821e606-b846-4c98-a7c9-ecde0395368c-kube-api-access-zwttk\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:15:14.347486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.347480 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:15:14.347486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.347489 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6821e606-b846-4c98-a7c9-ecde0395368c-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:15:14.896518 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.896440 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" Sep 10 19:15:14.896518 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.896435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0q5cfk" event={"ID":"6821e606-b846-4c98-a7c9-ecde0395368c","Type":"ContainerDied","Data":"a4efb346b313c0f9c6941a38f5dfdb3a09fb82aed8411afa9fc0baa6bb3cbd75"} Sep 10 19:15:14.896718 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.896558 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4efb346b313c0f9c6941a38f5dfdb3a09fb82aed8411afa9fc0baa6bb3cbd75" Sep 10 19:15:14.898213 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:14.898188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" event={"ID":"c2c03473-d443-4ee3-b9ad-18d41c71da27","Type":"ContainerStarted","Data":"56dee5590bc6ff96314bb9fc88ca7ac3c8922bad4bdda081d947f32bd747ed4d"} Sep 10 19:15:15.904733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:15.904696 2576 generic.go:356] "Generic (PLEG): container finished" podID="c2c03473-d443-4ee3-b9ad-18d41c71da27" containerID="56dee5590bc6ff96314bb9fc88ca7ac3c8922bad4bdda081d947f32bd747ed4d" exitCode=0 Sep 10 19:15:15.905112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:15.904752 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" event={"ID":"c2c03473-d443-4ee3-b9ad-18d41c71da27","Type":"ContainerDied","Data":"56dee5590bc6ff96314bb9fc88ca7ac3c8922bad4bdda081d947f32bd747ed4d"} Sep 10 19:15:16.910971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:16.910929 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" event={"ID":"c2c03473-d443-4ee3-b9ad-18d41c71da27","Type":"ContainerStarted","Data":"8f85e693c277f00b70accab585685948a34a9bf2899ab44b2951c4f240934d82"} Sep 10 19:15:16.911349 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:16.911114 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:16.951189 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:16.951129 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" podStartSLOduration=2.290231613 podStartE2EDuration="6.95111052s" totalImagesPullingTime="3.630716582s" totalInitContainerRuntime="1.030162325s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:16.94974562 +0000 UTC m=+282.866401649" watchObservedRunningTime="2026-09-10 19:15:16.95111052 +0000 UTC m=+282.867766549" Sep 10 19:15:21.883886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:21.883856 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-5q9jz" Sep 10 19:15:27.916953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:27.916921 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-gct5x" Sep 10 19:15:30.009062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:30.009003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:30.235750 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:30.235711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:30.635342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:30.635313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:30.911445 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:30.911343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:33.377874 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:33.377844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:33.537711 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:33.537675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:34.508873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:34.508838 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:15:34.510267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:34.510236 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:15:34.520914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:34.520883 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 19:15:34.994227 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:34.994188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:38.438517 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:38.438486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:39.691920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.691890 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52"] Sep 10 19:15:39.692277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.692172 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6821e606-b846-4c98-a7c9-ecde0395368c" containerName="util" Sep 10 19:15:39.692277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.692183 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6821e606-b846-4c98-a7c9-ecde0395368c" containerName="util" Sep 10 19:15:39.692277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.692196 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6821e606-b846-4c98-a7c9-ecde0395368c" containerName="pull" Sep 10 19:15:39.692277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.692201 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6821e606-b846-4c98-a7c9-ecde0395368c" containerName="pull" Sep 10 19:15:39.692277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.692211 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6821e606-b846-4c98-a7c9-ecde0395368c" containerName="extract" Sep 10 19:15:39.692277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.692217 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6821e606-b846-4c98-a7c9-ecde0395368c" containerName="extract" Sep 10 19:15:39.692277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.692267 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6821e606-b846-4c98-a7c9-ecde0395368c" containerName="extract" Sep 10 19:15:39.695301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.695285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.698481 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.698463 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b9qdh\"" Sep 10 19:15:39.699293 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.699279 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 19:15:39.717478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.717453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52"] Sep 10 19:15:39.753191 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753329 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753329 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753329 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llzbc\" (UniqueName: \"kubernetes.io/projected/e41275c9-a0cd-4f3c-b487-4510c5644384-kube-api-access-llzbc\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753329 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41275c9-a0cd-4f3c-b487-4510c5644384-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.753531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.753436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llzbc\" (UniqueName: \"kubernetes.io/projected/e41275c9-a0cd-4f3c-b487-4510c5644384-kube-api-access-llzbc\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41275c9-a0cd-4f3c-b487-4510c5644384-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854798 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854687 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854798 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.854906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.854884 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.855102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.855076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41275c9-a0cd-4f3c-b487-4510c5644384-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.857040 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.857020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.857202 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.857130 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.880825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.880800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41275c9-a0cd-4f3c-b487-4510c5644384-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:39.881029 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:39.881004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llzbc\" (UniqueName: \"kubernetes.io/projected/e41275c9-a0cd-4f3c-b487-4510c5644384-kube-api-access-llzbc\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cjlf52\" (UID: \"e41275c9-a0cd-4f3c-b487-4510c5644384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:40.005782 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:40.005698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:40.291688 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:40.291665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52"] Sep 10 19:15:40.293016 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:15:40.292991 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41275c9_a0cd_4f3c_b487_4510c5644384.slice/crio-b13108de671a639bd82ad84d6c45901faea8d7c8b91e1be9ec54672b48589efe WatchSource:0}: Error finding container b13108de671a639bd82ad84d6c45901faea8d7c8b91e1be9ec54672b48589efe: Status 404 returned error can't find the container with id b13108de671a639bd82ad84d6c45901faea8d7c8b91e1be9ec54672b48589efe Sep 10 19:15:40.294929 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:40.294913 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:15:40.352387 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:40.352336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:41.026633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.026599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" event={"ID":"e41275c9-a0cd-4f3c-b487-4510c5644384","Type":"ContainerStarted","Data":"b13108de671a639bd82ad84d6c45901faea8d7c8b91e1be9ec54672b48589efe"} Sep 10 19:15:41.171171 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.171139 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fpn29"] Sep 10 19:15:41.175790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.175770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:41.178695 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:15:41.178666 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"odh-model-controller-7998c67dc4-fpn29\" is forbidden: User \"system:node:ip-10-0-143-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-43.ec2.internal' and this object" podUID="ae3bb302-cb2b-436c-9df6-09384999fc94" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:41.179243 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:15:41.179217 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-143-43.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-43.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" Sep 10 19:15:41.179307 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:15:41.179217 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-dockercfg-r5q6f\" is forbidden: User \"system:node:ip-10-0-143-43.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-43.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r5q6f\"" type="*v1.Secret" Sep 10 19:15:41.200105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.200076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fpn29"] Sep 10 19:15:41.252022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.251983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:41.266352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.266321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae3bb302-cb2b-436c-9df6-09384999fc94-cert\") pod \"odh-model-controller-7998c67dc4-fpn29\" (UID: \"ae3bb302-cb2b-436c-9df6-09384999fc94\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:41.266476 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.266458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv42p\" (UniqueName: \"kubernetes.io/projected/ae3bb302-cb2b-436c-9df6-09384999fc94-kube-api-access-gv42p\") pod \"odh-model-controller-7998c67dc4-fpn29\" (UID: \"ae3bb302-cb2b-436c-9df6-09384999fc94\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:41.367401 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.367356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae3bb302-cb2b-436c-9df6-09384999fc94-cert\") pod \"odh-model-controller-7998c67dc4-fpn29\" (UID: \"ae3bb302-cb2b-436c-9df6-09384999fc94\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:41.367574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.367452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv42p\" (UniqueName: \"kubernetes.io/projected/ae3bb302-cb2b-436c-9df6-09384999fc94-kube-api-access-gv42p\") pod \"odh-model-controller-7998c67dc4-fpn29\" (UID: \"ae3bb302-cb2b-436c-9df6-09384999fc94\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:41.389920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.389877 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv42p\" (UniqueName: \"kubernetes.io/projected/ae3bb302-cb2b-436c-9df6-09384999fc94-kube-api-access-gv42p\") pod \"odh-model-controller-7998c67dc4-fpn29\" (UID: \"ae3bb302-cb2b-436c-9df6-09384999fc94\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:41.512649 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.512617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:41.689600 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:41.689501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:42.310826 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:42.310782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:42.330705 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:42.330637 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r5q6f\"" Sep 10 19:15:42.368647 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:15:42.368605 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: failed to sync secret cache: timed out waiting for the condition Sep 10 19:15:42.368835 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:15:42.368688 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3bb302-cb2b-436c-9df6-09384999fc94-cert podName:ae3bb302-cb2b-436c-9df6-09384999fc94 nodeName:}" failed. No retries permitted until 2026-09-10 19:15:42.868670499 +0000 UTC m=+308.785326517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ae3bb302-cb2b-436c-9df6-09384999fc94-cert") pod "odh-model-controller-7998c67dc4-fpn29" (UID: "ae3bb302-cb2b-436c-9df6-09384999fc94") : failed to sync secret cache: timed out waiting for the condition Sep 10 19:15:42.462200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:42.462166 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 19:15:42.881941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:42.881877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae3bb302-cb2b-436c-9df6-09384999fc94-cert\") pod \"odh-model-controller-7998c67dc4-fpn29\" (UID: \"ae3bb302-cb2b-436c-9df6-09384999fc94\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:42.884639 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:42.884616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae3bb302-cb2b-436c-9df6-09384999fc94-cert\") pod \"odh-model-controller-7998c67dc4-fpn29\" (UID: \"ae3bb302-cb2b-436c-9df6-09384999fc94\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:42.986753 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:42.986713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:43.006748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:43.006712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:43.859090 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:43.859043 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:15:43.859529 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:43.859143 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:15:43.859529 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:43.859180 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:15:44.046779 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:44.046692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" event={"ID":"e41275c9-a0cd-4f3c-b487-4510c5644384","Type":"ContainerStarted","Data":"b80485b925f0d852f271926d4b32bf2ea627391ea925b67d7a75f63ef81cd1ab"} Sep 10 19:15:44.072822 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:44.072395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fpn29"] Sep 10 19:15:44.075681 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:15:44.075640 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3bb302_cb2b_436c_9df6_09384999fc94.slice/crio-2cb890176f79f89853e5e69ce5d0ad3ac3df0367fb722967be0331a66dbead64 WatchSource:0}: Error finding container 2cb890176f79f89853e5e69ce5d0ad3ac3df0367fb722967be0331a66dbead64: Status 404 returned error can't find the container with id 2cb890176f79f89853e5e69ce5d0ad3ac3df0367fb722967be0331a66dbead64 Sep 10 19:15:44.127475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:44.127268 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" podStartSLOduration=1.5635236460000002 podStartE2EDuration="5.127247466s" totalImagesPullingTime="3.56372382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:44.12547269 +0000 UTC m=+310.042128718" watchObservedRunningTime="2026-09-10 19:15:44.127247466 +0000 UTC m=+310.043903497" Sep 10 19:15:45.006301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:45.006260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:45.012023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:45.011998 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:45.055657 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:45.055408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" event={"ID":"ae3bb302-cb2b-436c-9df6-09384999fc94","Type":"ContainerStarted","Data":"2cb890176f79f89853e5e69ce5d0ad3ac3df0367fb722967be0331a66dbead64"} Sep 10 19:15:45.056065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:45.055992 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:45.057217 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:45.057196 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cjlf52" Sep 10 19:15:49.080752 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.080710 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" event={"ID":"ae3bb302-cb2b-436c-9df6-09384999fc94","Type":"ContainerStarted","Data":"e3c0cf416808cf831347380cf7c5151a94db0549e724ab08f3b5340760ee5bd8"} Sep 10 19:15:49.081188 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.080829 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:15:49.111522 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.111477 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" podStartSLOduration=3.44314915 podStartE2EDuration="8.11146254s" totalImagesPullingTime="4.66831339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:49.109894645 +0000 UTC m=+315.026550688" watchObservedRunningTime="2026-09-10 19:15:49.11146254 +0000 UTC m=+315.028118568" Sep 10 19:15:49.814726 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.814678 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9"] Sep 10 19:15:49.817492 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.817474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.822784 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.822742 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 19:15:49.822784 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.822749 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 19:15:49.822784 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.822749 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 19:15:49.823119 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.822838 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z4h2z\"" Sep 10 19:15:49.840061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.840037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.840195 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.840079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mf6\" (UniqueName: \"kubernetes.io/projected/06dc50da-ff3e-4041-a38d-051cc83baf6f-kube-api-access-l7mf6\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.840195 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.840099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.840195 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.840116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.840195 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.840191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.840425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.840265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.840425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.840329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dc50da-ff3e-4041-a38d-051cc83baf6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.861120 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.861091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9"] Sep 10 19:15:49.940759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.940722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dc50da-ff3e-4041-a38d-051cc83baf6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.940922 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.940769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.940922 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.940809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mf6\" (UniqueName: \"kubernetes.io/projected/06dc50da-ff3e-4041-a38d-051cc83baf6f-kube-api-access-l7mf6\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.940922 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.940912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.941083 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.940945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.941131 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.941098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.941183 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.941163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.941791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.941761 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.943282 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.943254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.943409 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.943300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.943609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.943592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dc50da-ff3e-4041-a38d-051cc83baf6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.943755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.943738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.950602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.950581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dc50da-ff3e-4041-a38d-051cc83baf6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:49.954289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:49.954268 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mf6\" (UniqueName: \"kubernetes.io/projected/06dc50da-ff3e-4041-a38d-051cc83baf6f-kube-api-access-l7mf6\") pod \"istiod-openshift-gateway-5677ccb49b-lswj9\" (UID: \"06dc50da-ff3e-4041-a38d-051cc83baf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:50.125883 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:50.125797 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:50.426038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:50.425943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9"] Sep 10 19:15:50.428120 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:15:50.428096 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dc50da_ff3e_4041_a38d_051cc83baf6f.slice/crio-a6e43eec62d5e4b057e6e8f6d97c0b76c3c32da320b7697ac8b08fb59dc84d50 WatchSource:0}: Error finding container a6e43eec62d5e4b057e6e8f6d97c0b76c3c32da320b7697ac8b08fb59dc84d50: Status 404 returned error can't find the container with id a6e43eec62d5e4b057e6e8f6d97c0b76c3c32da320b7697ac8b08fb59dc84d50 Sep 10 19:15:51.092439 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:51.092400 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" event={"ID":"06dc50da-ff3e-4041-a38d-051cc83baf6f","Type":"ContainerStarted","Data":"a6e43eec62d5e4b057e6e8f6d97c0b76c3c32da320b7697ac8b08fb59dc84d50"} Sep 10 19:15:52.748785 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:52.748741 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:15:52.749124 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:52.748825 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:15:53.104957 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:53.104926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" event={"ID":"06dc50da-ff3e-4041-a38d-051cc83baf6f","Type":"ContainerStarted","Data":"9767d5f87dac87ff82f53bf30e51cba4e7220b69bc020e871962b5ad235f2b0e"} Sep 10 19:15:53.105138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:53.105094 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:15:53.106604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:53.106579 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-lswj9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 19:15:53.106727 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:53.106633 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" podUID="06dc50da-ff3e-4041-a38d-051cc83baf6f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 19:15:53.283187 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:53.283137 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" podStartSLOduration=1.9647972999999999 podStartE2EDuration="4.283120313s" totalImagesPullingTime="2.318323013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:53.26167764 +0000 UTC m=+319.178333669" watchObservedRunningTime="2026-09-10 19:15:53.283120313 +0000 UTC m=+319.199776342" Sep 10 19:15:54.110467 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:54.110432 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-lswj9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 19:15:54.110815 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:54.110500 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" podUID="06dc50da-ff3e-4041-a38d-051cc83baf6f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 19:15:54.510194 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:54.510108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:54.881396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:54.881333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:55.031311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:55.031275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:55.208029 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:55.207925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:55.367308 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:55.367281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:15:57.111211 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:15:57.111176 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lswj9" Sep 10 19:16:00.088152 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:00.088119 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fpn29" Sep 10 19:16:42.116132 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:42.116056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:42.217592 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:42.217560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:43.569072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:43.569043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:43.670618 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:43.670590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:45.494386 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:45.494338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:45.564914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:45.564881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:46.362882 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.362852 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n"] Sep 10 19:16:46.371082 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.371060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.373683 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.373661 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 19:16:46.373812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.373659 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ks6mr\"" Sep 10 19:16:46.374614 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.374594 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 19:16:46.379532 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.379506 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n"] Sep 10 19:16:46.416918 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.416887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxff\" (UniqueName: \"kubernetes.io/projected/1bf45b00-eda1-4849-85e3-10e11f6ca57e-kube-api-access-qgxff\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.417073 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.416945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.417073 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.417016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.453780 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.453755 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9"] Sep 10 19:16:46.457633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.457618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.466712 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.466685 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9"] Sep 10 19:16:46.518338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxff\" (UniqueName: \"kubernetes.io/projected/1bf45b00-eda1-4849-85e3-10e11f6ca57e-kube-api-access-qgxff\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.518732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfvsf\" (UniqueName: \"kubernetes.io/projected/a4447b28-677f-45f0-b513-7541a1a36490-kube-api-access-bfvsf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.518732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.518732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.518732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.518732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.518899 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.518899 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.518893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.528124 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.528097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxff\" (UniqueName: \"kubernetes.io/projected/1bf45b00-eda1-4849-85e3-10e11f6ca57e-kube-api-access-qgxff\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.559509 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.559482 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj"] Sep 10 19:16:46.563361 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.563343 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.574261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.574242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj"] Sep 10 19:16:46.619005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.618928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc2sp\" (UniqueName: \"kubernetes.io/projected/b1edc212-9806-4f74-9ae7-875c99c56fe3-kube-api-access-jc2sp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.619005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.618975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.619200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.619011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.619200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.619037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.619200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.619079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfvsf\" (UniqueName: \"kubernetes.io/projected/a4447b28-677f-45f0-b513-7541a1a36490-kube-api-access-bfvsf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.619200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.619127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.619464 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.619328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.619513 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.619458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.628256 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.628222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfvsf\" (UniqueName: \"kubernetes.io/projected/a4447b28-677f-45f0-b513-7541a1a36490-kube-api-access-bfvsf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.658024 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.657992 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5"] Sep 10 19:16:46.662007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.661987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.673800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.673778 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5"] Sep 10 19:16:46.681564 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.681541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:46.720159 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.720125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvw6\" (UniqueName: \"kubernetes.io/projected/2d6929de-cbd4-449c-92d3-223f1c206359-kube-api-access-wwvw6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.720515 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.720492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.720658 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.720639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc2sp\" (UniqueName: \"kubernetes.io/projected/b1edc212-9806-4f74-9ae7-875c99c56fe3-kube-api-access-jc2sp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.720736 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.720684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.720789 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.720739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.720789 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.720763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.721171 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.721147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.721321 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.721300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.732476 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.732203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc2sp\" (UniqueName: \"kubernetes.io/projected/b1edc212-9806-4f74-9ae7-875c99c56fe3-kube-api-access-jc2sp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.767101 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.767061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:46.821239 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.821203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.821438 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.821330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.821514 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.821495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvw6\" (UniqueName: \"kubernetes.io/projected/2d6929de-cbd4-449c-92d3-223f1c206359-kube-api-access-wwvw6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.821713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.821676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.821873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.821844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.831924 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.831896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvw6\" (UniqueName: \"kubernetes.io/projected/2d6929de-cbd4-449c-92d3-223f1c206359-kube-api-access-wwvw6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:46.882604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.882503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:46.966519 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.965570 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n"] Sep 10 19:16:46.970246 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:16:46.970213 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf45b00_eda1_4849_85e3_10e11f6ca57e.slice/crio-18c34bb2dc9b38d16a6f41cbb308012d216319df4142b429eab36f711050be37 WatchSource:0}: Error finding container 18c34bb2dc9b38d16a6f41cbb308012d216319df4142b429eab36f711050be37: Status 404 returned error can't find the container with id 18c34bb2dc9b38d16a6f41cbb308012d216319df4142b429eab36f711050be37 Sep 10 19:16:46.971036 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:46.970945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:47.057673 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.057647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9"] Sep 10 19:16:47.059542 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:16:47.059511 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4447b28_677f_45f0_b513_7541a1a36490.slice/crio-93b14eeb4002d01695fd0ed22771727bafc473ce2506146edce9744158e6dcb9 WatchSource:0}: Error finding container 93b14eeb4002d01695fd0ed22771727bafc473ce2506146edce9744158e6dcb9: Status 404 returned error can't find the container with id 93b14eeb4002d01695fd0ed22771727bafc473ce2506146edce9744158e6dcb9 Sep 10 19:16:47.166842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.166814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj"] Sep 10 19:16:47.173195 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:16:47.173168 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1edc212_9806_4f74_9ae7_875c99c56fe3.slice/crio-f243869685d1289a71720a028c5e0fa48be48c1a66ea6dcb718fb734045773f3 WatchSource:0}: Error finding container f243869685d1289a71720a028c5e0fa48be48c1a66ea6dcb718fb734045773f3: Status 404 returned error can't find the container with id f243869685d1289a71720a028c5e0fa48be48c1a66ea6dcb718fb734045773f3 Sep 10 19:16:47.258491 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.258465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5"] Sep 10 19:16:47.280241 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:16:47.280211 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6929de_cbd4_449c_92d3_223f1c206359.slice/crio-36852f40eeefa30fb8e9a02628ac1eedff6af17d16415a6f2acb84cf57dee015 WatchSource:0}: Error finding container 36852f40eeefa30fb8e9a02628ac1eedff6af17d16415a6f2acb84cf57dee015: Status 404 returned error can't find the container with id 36852f40eeefa30fb8e9a02628ac1eedff6af17d16415a6f2acb84cf57dee015 Sep 10 19:16:47.367978 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.367944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" event={"ID":"2d6929de-cbd4-449c-92d3-223f1c206359","Type":"ContainerStarted","Data":"36852f40eeefa30fb8e9a02628ac1eedff6af17d16415a6f2acb84cf57dee015"} Sep 10 19:16:47.369540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.369518 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4447b28-677f-45f0-b513-7541a1a36490" containerID="3e3c690a48a7fcbadf7865084820f3c4b4da68bb69dc8167165df5df1c91a3f8" exitCode=0 Sep 10 19:16:47.369660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.369600 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" event={"ID":"a4447b28-677f-45f0-b513-7541a1a36490","Type":"ContainerDied","Data":"3e3c690a48a7fcbadf7865084820f3c4b4da68bb69dc8167165df5df1c91a3f8"} Sep 10 19:16:47.369660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.369648 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" event={"ID":"a4447b28-677f-45f0-b513-7541a1a36490","Type":"ContainerStarted","Data":"93b14eeb4002d01695fd0ed22771727bafc473ce2506146edce9744158e6dcb9"} Sep 10 19:16:47.371199 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.371131 2576 generic.go:356] "Generic (PLEG): container finished" podID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerID="6508a386412531c770cf606497fdabf327680c6da46500bc34089fd8dd59c391" exitCode=0 Sep 10 19:16:47.371255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.371206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" event={"ID":"b1edc212-9806-4f74-9ae7-875c99c56fe3","Type":"ContainerDied","Data":"6508a386412531c770cf606497fdabf327680c6da46500bc34089fd8dd59c391"} Sep 10 19:16:47.371255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.371242 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" event={"ID":"b1edc212-9806-4f74-9ae7-875c99c56fe3","Type":"ContainerStarted","Data":"f243869685d1289a71720a028c5e0fa48be48c1a66ea6dcb718fb734045773f3"} Sep 10 19:16:47.372939 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.372919 2576 generic.go:356] "Generic (PLEG): container finished" podID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerID="ae59d528bb1da509843743bbabdae112ec292671ba9a2aedc2728b38cdfea2ce" exitCode=0 Sep 10 19:16:47.373033 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.372958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" event={"ID":"1bf45b00-eda1-4849-85e3-10e11f6ca57e","Type":"ContainerDied","Data":"ae59d528bb1da509843743bbabdae112ec292671ba9a2aedc2728b38cdfea2ce"} Sep 10 19:16:47.373033 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:47.372994 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" event={"ID":"1bf45b00-eda1-4849-85e3-10e11f6ca57e","Type":"ContainerStarted","Data":"18c34bb2dc9b38d16a6f41cbb308012d216319df4142b429eab36f711050be37"} Sep 10 19:16:48.381240 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:48.381203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" event={"ID":"b1edc212-9806-4f74-9ae7-875c99c56fe3","Type":"ContainerStarted","Data":"a066bd866adb6d73ac07bf49b5f5dd206cebf952c81de7636d09f793562fdc06"} Sep 10 19:16:48.384245 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:48.383862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" event={"ID":"1bf45b00-eda1-4849-85e3-10e11f6ca57e","Type":"ContainerStarted","Data":"e919852934ea3093fab081dca971a244f392b0f3efa4ab9c28625928bac30715"} Sep 10 19:16:48.385941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:48.385915 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d6929de-cbd4-449c-92d3-223f1c206359" containerID="07d46ad870abde498d66f5797a67dc16871da345399e416e257df80a70125a32" exitCode=0 Sep 10 19:16:48.386062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:48.385970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" event={"ID":"2d6929de-cbd4-449c-92d3-223f1c206359","Type":"ContainerDied","Data":"07d46ad870abde498d66f5797a67dc16871da345399e416e257df80a70125a32"} Sep 10 19:16:49.393996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:49.393968 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4447b28-677f-45f0-b513-7541a1a36490" containerID="997b25e26bf1c7c178f50b9f16a4cc1e6e17af1b6f0f0388ee692fdaf511e0ef" exitCode=0 Sep 10 19:16:49.394393 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:49.394017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" event={"ID":"a4447b28-677f-45f0-b513-7541a1a36490","Type":"ContainerDied","Data":"997b25e26bf1c7c178f50b9f16a4cc1e6e17af1b6f0f0388ee692fdaf511e0ef"} Sep 10 19:16:49.396061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:49.396041 2576 generic.go:356] "Generic (PLEG): container finished" podID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerID="a066bd866adb6d73ac07bf49b5f5dd206cebf952c81de7636d09f793562fdc06" exitCode=0 Sep 10 19:16:49.396153 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:49.396107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" event={"ID":"b1edc212-9806-4f74-9ae7-875c99c56fe3","Type":"ContainerDied","Data":"a066bd866adb6d73ac07bf49b5f5dd206cebf952c81de7636d09f793562fdc06"} Sep 10 19:16:49.398046 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:49.398023 2576 generic.go:356] "Generic (PLEG): container finished" podID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerID="e919852934ea3093fab081dca971a244f392b0f3efa4ab9c28625928bac30715" exitCode=0 Sep 10 19:16:49.398143 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:49.398093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" event={"ID":"1bf45b00-eda1-4849-85e3-10e11f6ca57e","Type":"ContainerDied","Data":"e919852934ea3093fab081dca971a244f392b0f3efa4ab9c28625928bac30715"} Sep 10 19:16:50.405182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.405147 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4447b28-677f-45f0-b513-7541a1a36490" containerID="0d6fac5fd26886315d7954d0fb42e799e871555ab9956d1e4c7d9f0617bb478c" exitCode=0 Sep 10 19:16:50.405584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.405229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" event={"ID":"a4447b28-677f-45f0-b513-7541a1a36490","Type":"ContainerDied","Data":"0d6fac5fd26886315d7954d0fb42e799e871555ab9956d1e4c7d9f0617bb478c"} Sep 10 19:16:50.407352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.407326 2576 generic.go:356] "Generic (PLEG): container finished" podID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerID="1a55cf49e394e77a700b16f042732cf602006215af2f29c964333ab4f3cce422" exitCode=0 Sep 10 19:16:50.407480 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.407430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" event={"ID":"b1edc212-9806-4f74-9ae7-875c99c56fe3","Type":"ContainerDied","Data":"1a55cf49e394e77a700b16f042732cf602006215af2f29c964333ab4f3cce422"} Sep 10 19:16:50.409358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.409335 2576 generic.go:356] "Generic (PLEG): container finished" podID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerID="59434ba2b8b43aebd940de25bf500908de5c536410da418b88ded52ed3c6c11f" exitCode=0 Sep 10 19:16:50.409463 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.409424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" event={"ID":"1bf45b00-eda1-4849-85e3-10e11f6ca57e","Type":"ContainerDied","Data":"59434ba2b8b43aebd940de25bf500908de5c536410da418b88ded52ed3c6c11f"} Sep 10 19:16:50.411258 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.411236 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d6929de-cbd4-449c-92d3-223f1c206359" containerID="28e39a487a16d72a6af2c62d3f9ffb2c0a0e300b3f6c86a61e70e7e47adaaa5b" exitCode=0 Sep 10 19:16:50.411327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:50.411287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" event={"ID":"2d6929de-cbd4-449c-92d3-223f1c206359","Type":"ContainerDied","Data":"28e39a487a16d72a6af2c62d3f9ffb2c0a0e300b3f6c86a61e70e7e47adaaa5b"} Sep 10 19:16:51.418430 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.418394 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d6929de-cbd4-449c-92d3-223f1c206359" containerID="d773d5802d59cda5c19b72225f46e76c482c3b6c76fcaab921850ff7a52ed494" exitCode=0 Sep 10 19:16:51.418807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.418518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" event={"ID":"2d6929de-cbd4-449c-92d3-223f1c206359","Type":"ContainerDied","Data":"d773d5802d59cda5c19b72225f46e76c482c3b6c76fcaab921850ff7a52ed494"} Sep 10 19:16:51.610072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.610052 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:51.613149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.613130 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:51.616385 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.616353 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:51.662520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662493 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-bundle\") pod \"b1edc212-9806-4f74-9ae7-875c99c56fe3\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " Sep 10 19:16:51.662687 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662540 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-bundle\") pod \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " Sep 10 19:16:51.662687 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-bundle\") pod \"a4447b28-677f-45f0-b513-7541a1a36490\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " Sep 10 19:16:51.662687 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662613 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgxff\" (UniqueName: \"kubernetes.io/projected/1bf45b00-eda1-4849-85e3-10e11f6ca57e-kube-api-access-qgxff\") pod \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " Sep 10 19:16:51.662687 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662643 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-util\") pod \"b1edc212-9806-4f74-9ae7-875c99c56fe3\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " Sep 10 19:16:51.662687 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-util\") pod \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\" (UID: \"1bf45b00-eda1-4849-85e3-10e11f6ca57e\") " Sep 10 19:16:51.662940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662704 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfvsf\" (UniqueName: \"kubernetes.io/projected/a4447b28-677f-45f0-b513-7541a1a36490-kube-api-access-bfvsf\") pod \"a4447b28-677f-45f0-b513-7541a1a36490\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " Sep 10 19:16:51.662940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc2sp\" (UniqueName: \"kubernetes.io/projected/b1edc212-9806-4f74-9ae7-875c99c56fe3-kube-api-access-jc2sp\") pod \"b1edc212-9806-4f74-9ae7-875c99c56fe3\" (UID: \"b1edc212-9806-4f74-9ae7-875c99c56fe3\") " Sep 10 19:16:51.662940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.662773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-util\") pod \"a4447b28-677f-45f0-b513-7541a1a36490\" (UID: \"a4447b28-677f-45f0-b513-7541a1a36490\") " Sep 10 19:16:51.664421 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.664301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-bundle" (OuterVolumeSpecName: "bundle") pod "a4447b28-677f-45f0-b513-7541a1a36490" (UID: "a4447b28-677f-45f0-b513-7541a1a36490"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:51.664567 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.664520 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-bundle" (OuterVolumeSpecName: "bundle") pod "1bf45b00-eda1-4849-85e3-10e11f6ca57e" (UID: "1bf45b00-eda1-4849-85e3-10e11f6ca57e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:51.665909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.665093 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-bundle" (OuterVolumeSpecName: "bundle") pod "b1edc212-9806-4f74-9ae7-875c99c56fe3" (UID: "b1edc212-9806-4f74-9ae7-875c99c56fe3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:51.669968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.669895 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1edc212-9806-4f74-9ae7-875c99c56fe3-kube-api-access-jc2sp" (OuterVolumeSpecName: "kube-api-access-jc2sp") pod "b1edc212-9806-4f74-9ae7-875c99c56fe3" (UID: "b1edc212-9806-4f74-9ae7-875c99c56fe3"). InnerVolumeSpecName "kube-api-access-jc2sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:16:51.670277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.670254 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4447b28-677f-45f0-b513-7541a1a36490-kube-api-access-bfvsf" (OuterVolumeSpecName: "kube-api-access-bfvsf") pod "a4447b28-677f-45f0-b513-7541a1a36490" (UID: "a4447b28-677f-45f0-b513-7541a1a36490"). InnerVolumeSpecName "kube-api-access-bfvsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:16:51.670390 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.670310 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-util" (OuterVolumeSpecName: "util") pod "a4447b28-677f-45f0-b513-7541a1a36490" (UID: "a4447b28-677f-45f0-b513-7541a1a36490"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:51.672842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.672815 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf45b00-eda1-4849-85e3-10e11f6ca57e-kube-api-access-qgxff" (OuterVolumeSpecName: "kube-api-access-qgxff") pod "1bf45b00-eda1-4849-85e3-10e11f6ca57e" (UID: "1bf45b00-eda1-4849-85e3-10e11f6ca57e"). InnerVolumeSpecName "kube-api-access-qgxff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:16:51.677453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.677425 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-util" (OuterVolumeSpecName: "util") pod "b1edc212-9806-4f74-9ae7-875c99c56fe3" (UID: "b1edc212-9806-4f74-9ae7-875c99c56fe3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:51.678255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.678229 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-util" (OuterVolumeSpecName: "util") pod "1bf45b00-eda1-4849-85e3-10e11f6ca57e" (UID: "1bf45b00-eda1-4849-85e3-10e11f6ca57e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:51.763459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763422 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763455 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763463 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763473 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxff\" (UniqueName: \"kubernetes.io/projected/1bf45b00-eda1-4849-85e3-10e11f6ca57e-kube-api-access-qgxff\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763482 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1edc212-9806-4f74-9ae7-875c99c56fe3-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763490 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bf45b00-eda1-4849-85e3-10e11f6ca57e-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763497 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfvsf\" (UniqueName: \"kubernetes.io/projected/a4447b28-677f-45f0-b513-7541a1a36490-kube-api-access-bfvsf\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763505 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc2sp\" (UniqueName: \"kubernetes.io/projected/b1edc212-9806-4f74-9ae7-875c99c56fe3-kube-api-access-jc2sp\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:51.763680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:51.763514 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4447b28-677f-45f0-b513-7541a1a36490-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:52.424930 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.424878 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" Sep 10 19:16:52.425424 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.424896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adlcl9" event={"ID":"a4447b28-677f-45f0-b513-7541a1a36490","Type":"ContainerDied","Data":"93b14eeb4002d01695fd0ed22771727bafc473ce2506146edce9744158e6dcb9"} Sep 10 19:16:52.425424 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.425035 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b14eeb4002d01695fd0ed22771727bafc473ce2506146edce9744158e6dcb9" Sep 10 19:16:52.426734 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.426711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" event={"ID":"b1edc212-9806-4f74-9ae7-875c99c56fe3","Type":"ContainerDied","Data":"f243869685d1289a71720a028c5e0fa48be48c1a66ea6dcb718fb734045773f3"} Sep 10 19:16:52.426734 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.426726 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csxllj" Sep 10 19:16:52.426895 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.426750 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f243869685d1289a71720a028c5e0fa48be48c1a66ea6dcb718fb734045773f3" Sep 10 19:16:52.428671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.428640 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" event={"ID":"1bf45b00-eda1-4849-85e3-10e11f6ca57e","Type":"ContainerDied","Data":"18c34bb2dc9b38d16a6f41cbb308012d216319df4142b429eab36f711050be37"} Sep 10 19:16:52.428671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.428660 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7b9q5n" Sep 10 19:16:52.428818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.428685 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18c34bb2dc9b38d16a6f41cbb308012d216319df4142b429eab36f711050be37" Sep 10 19:16:52.556068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.556046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:52.570096 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.570077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwvw6\" (UniqueName: \"kubernetes.io/projected/2d6929de-cbd4-449c-92d3-223f1c206359-kube-api-access-wwvw6\") pod \"2d6929de-cbd4-449c-92d3-223f1c206359\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " Sep 10 19:16:52.570183 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.570138 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-bundle\") pod \"2d6929de-cbd4-449c-92d3-223f1c206359\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " Sep 10 19:16:52.570233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.570200 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-util\") pod \"2d6929de-cbd4-449c-92d3-223f1c206359\" (UID: \"2d6929de-cbd4-449c-92d3-223f1c206359\") " Sep 10 19:16:52.570615 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.570592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-bundle" (OuterVolumeSpecName: "bundle") pod "2d6929de-cbd4-449c-92d3-223f1c206359" (UID: "2d6929de-cbd4-449c-92d3-223f1c206359"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:52.572186 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.572164 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6929de-cbd4-449c-92d3-223f1c206359-kube-api-access-wwvw6" (OuterVolumeSpecName: "kube-api-access-wwvw6") pod "2d6929de-cbd4-449c-92d3-223f1c206359" (UID: "2d6929de-cbd4-449c-92d3-223f1c206359"). InnerVolumeSpecName "kube-api-access-wwvw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:16:52.641623 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.641572 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-util" (OuterVolumeSpecName: "util") pod "2d6929de-cbd4-449c-92d3-223f1c206359" (UID: "2d6929de-cbd4-449c-92d3-223f1c206359"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:16:52.671934 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.671903 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:52.671934 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.671927 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d6929de-cbd4-449c-92d3-223f1c206359-util\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:52.671934 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:52.671936 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvw6\" (UniqueName: \"kubernetes.io/projected/2d6929de-cbd4-449c-92d3-223f1c206359-kube-api-access-wwvw6\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:16:53.435018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:53.434993 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" Sep 10 19:16:53.435419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:53.434988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nd5m5" event={"ID":"2d6929de-cbd4-449c-92d3-223f1c206359","Type":"ContainerDied","Data":"36852f40eeefa30fb8e9a02628ac1eedff6af17d16415a6f2acb84cf57dee015"} Sep 10 19:16:53.435419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:53.435115 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36852f40eeefa30fb8e9a02628ac1eedff6af17d16415a6f2acb84cf57dee015" Sep 10 19:16:54.925364 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:54.925332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:55.732182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:55.732142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:56.563909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:56.563875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:59.815829 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:59.815792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:16:59.997550 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:16:59.997510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:00.190602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:00.190518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:00.409860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:00.409831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:00.852867 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:00.852836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:03.433038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:03.433004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:05.423789 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:05.423755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:05.787448 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:05.787418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:06.297781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:06.297753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:06.443851 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:06.443819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:08.222725 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:08.222685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:09.736555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:09.736521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:09.998832 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:09.998747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:10.340576 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:10.340542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:10.517531 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:10.517500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:11.388421 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:11.388387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:11.493147 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:11.493115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:16.917637 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:16.917598 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8678bcdcfb-n64x4"] Sep 10 19:17:17.036264 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:17.036235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:17.138601 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:17.138566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:17.237487 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:17.237402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:20.462778 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:20.462748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:20.569598 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:20.569569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:23.395543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:23.395513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:23.579980 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:23.579950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:25.802305 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:25.802272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:25.871517 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:25.871488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:32.951284 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:32.951241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:33.032499 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:33.032460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:41.361225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:41.361181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:41.462083 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:41.462053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:41.939502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:41.939459 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8678bcdcfb-n64x4" podUID="aa1bd7eb-4971-4006-b359-dcac93f39ecd" containerName="console" containerID="cri-o://736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43" gracePeriod=15 Sep 10 19:17:42.187007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.186983 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:17:42.306301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306273 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-serving-cert\") pod \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " Sep 10 19:17:42.306497 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306320 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-trusted-ca-bundle\") pod \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " Sep 10 19:17:42.306497 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306350 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82w5k\" (UniqueName: \"kubernetes.io/projected/aa1bd7eb-4971-4006-b359-dcac93f39ecd-kube-api-access-82w5k\") pod \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " Sep 10 19:17:42.306497 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306474 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-service-ca\") pod \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " Sep 10 19:17:42.306677 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-config\") pod \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " Sep 10 19:17:42.306677 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306606 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-oauth-serving-cert\") pod \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " Sep 10 19:17:42.306677 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-oauth-config\") pod \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\" (UID: \"aa1bd7eb-4971-4006-b359-dcac93f39ecd\") " Sep 10 19:17:42.306824 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306744 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aa1bd7eb-4971-4006-b359-dcac93f39ecd" (UID: "aa1bd7eb-4971-4006-b359-dcac93f39ecd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:17:42.306921 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306893 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa1bd7eb-4971-4006-b359-dcac93f39ecd" (UID: "aa1bd7eb-4971-4006-b359-dcac93f39ecd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:17:42.306990 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306917 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-trusted-ca-bundle\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:17:42.306990 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306943 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-config" (OuterVolumeSpecName: "console-config") pod "aa1bd7eb-4971-4006-b359-dcac93f39ecd" (UID: "aa1bd7eb-4971-4006-b359-dcac93f39ecd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:17:42.306990 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.306960 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa1bd7eb-4971-4006-b359-dcac93f39ecd" (UID: "aa1bd7eb-4971-4006-b359-dcac93f39ecd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:17:42.308653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.308630 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa1bd7eb-4971-4006-b359-dcac93f39ecd" (UID: "aa1bd7eb-4971-4006-b359-dcac93f39ecd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:17:42.308773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.308703 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1bd7eb-4971-4006-b359-dcac93f39ecd-kube-api-access-82w5k" (OuterVolumeSpecName: "kube-api-access-82w5k") pod "aa1bd7eb-4971-4006-b359-dcac93f39ecd" (UID: "aa1bd7eb-4971-4006-b359-dcac93f39ecd"). InnerVolumeSpecName "kube-api-access-82w5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:17:42.308883 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.308858 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa1bd7eb-4971-4006-b359-dcac93f39ecd" (UID: "aa1bd7eb-4971-4006-b359-dcac93f39ecd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:17:42.408105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.408076 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82w5k\" (UniqueName: \"kubernetes.io/projected/aa1bd7eb-4971-4006-b359-dcac93f39ecd-kube-api-access-82w5k\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:17:42.408105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.408099 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:17:42.408105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.408108 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-config\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:17:42.408569 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.408116 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa1bd7eb-4971-4006-b359-dcac93f39ecd-oauth-serving-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:17:42.408569 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.408125 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-oauth-config\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:17:42.408569 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.408134 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1bd7eb-4971-4006-b359-dcac93f39ecd-console-serving-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:17:42.682585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.682549 2576 generic.go:356] "Generic (PLEG): container finished" podID="aa1bd7eb-4971-4006-b359-dcac93f39ecd" containerID="736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43" exitCode=0 Sep 10 19:17:42.682585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.682588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678bcdcfb-n64x4" event={"ID":"aa1bd7eb-4971-4006-b359-dcac93f39ecd","Type":"ContainerDied","Data":"736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43"} Sep 10 19:17:42.682781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.682595 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8678bcdcfb-n64x4" Sep 10 19:17:42.682781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.682623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8678bcdcfb-n64x4" event={"ID":"aa1bd7eb-4971-4006-b359-dcac93f39ecd","Type":"ContainerDied","Data":"30fe4ea8da750ff7d1c2f8bf3e0d073de1478460760b68846ec58ce74e2d5e57"} Sep 10 19:17:42.682781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.682642 2576 scope.go:117] "RemoveContainer" containerID="736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43" Sep 10 19:17:42.693756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.693734 2576 scope.go:117] "RemoveContainer" containerID="736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43" Sep 10 19:17:42.694033 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:17:42.694012 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43\": container with ID starting with 736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43 not found: ID does not exist" containerID="736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43" Sep 10 19:17:42.694082 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.694041 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43"} err="failed to get container status \"736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43\": rpc error: code = NotFound desc = could not find container \"736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43\": container with ID starting with 736bb79881a50453f57b4086f00c32b821215406dbd862aaf222b206a28dba43 not found: ID does not exist" Sep 10 19:17:42.700992 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.700948 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8678bcdcfb-n64x4"] Sep 10 19:17:42.702913 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:42.702893 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8678bcdcfb-n64x4"] Sep 10 19:17:44.601518 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:44.601483 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa1bd7eb-4971-4006-b359-dcac93f39ecd" path="/var/lib/kubelet/pods/aa1bd7eb-4971-4006-b359-dcac93f39ecd/volumes" Sep 10 19:17:44.822115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:44.822078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:44.923839 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:44.923756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:47.857970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:47.857940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:47.955097 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:47.955060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:49.486165 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:49.486135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:49.697957 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:49.697926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:54.058844 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:54.058811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:54.168577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:54.168543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:54.237172 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:54.237135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:54.921922 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:54.921890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:55.003119 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:55.003080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:55.100082 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:55.100054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:55.167707 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:55.167677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:55.616566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:55.616531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:55.738506 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:55.738475 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:56.221157 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:56.221116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:17:56.284295 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:17:56.284258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:04.823133 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:04.823096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:04.913615 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:04.913579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:05.025620 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:05.025557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:05.111924 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:05.111842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:07.454044 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:07.454006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:07.540791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:07.540761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:09.977648 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:09.977619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:10.067485 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:10.067456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:10.225562 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:10.225530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:10.318734 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:10.318697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:10.716221 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:10.716141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:23.517012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:23.516978 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:23.631993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:23.631933 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:31.981937 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:31.981900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:32.043050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043015 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79d7459bb-tfr5x"] Sep 10 19:18:32.043434 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043412 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa1bd7eb-4971-4006-b359-dcac93f39ecd" containerName="console" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043436 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa1bd7eb-4971-4006-b359-dcac93f39ecd" containerName="console" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043450 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerName="util" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043459 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerName="util" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043469 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4447b28-677f-45f0-b513-7541a1a36490" containerName="extract" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043478 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4447b28-677f-45f0-b513-7541a1a36490" containerName="extract" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043492 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerName="pull" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043500 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerName="pull" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043511 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerName="util" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043516 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerName="util" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043530 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerName="extract" Sep 10 19:18:32.043540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043536 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043550 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6929de-cbd4-449c-92d3-223f1c206359" containerName="util" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043558 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6929de-cbd4-449c-92d3-223f1c206359" containerName="util" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043568 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043576 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043590 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4447b28-677f-45f0-b513-7541a1a36490" containerName="util" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043598 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4447b28-677f-45f0-b513-7541a1a36490" containerName="util" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043606 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerName="pull" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043614 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerName="pull" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043622 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4447b28-677f-45f0-b513-7541a1a36490" containerName="pull" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043630 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4447b28-677f-45f0-b513-7541a1a36490" containerName="pull" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043639 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6929de-cbd4-449c-92d3-223f1c206359" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043646 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6929de-cbd4-449c-92d3-223f1c206359" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043656 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6929de-cbd4-449c-92d3-223f1c206359" containerName="pull" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043663 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6929de-cbd4-449c-92d3-223f1c206359" containerName="pull" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043756 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa1bd7eb-4971-4006-b359-dcac93f39ecd" containerName="console" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043772 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d6929de-cbd4-449c-92d3-223f1c206359" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043780 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf45b00-eda1-4849-85e3-10e11f6ca57e" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043788 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4447b28-677f-45f0-b513-7541a1a36490" containerName="extract" Sep 10 19:18:32.044047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.043798 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1edc212-9806-4f74-9ae7-875c99c56fe3" containerName="extract" Sep 10 19:18:32.046724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.046707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.050635 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.050612 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 19:18:32.050815 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.050644 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ltnwd\"" Sep 10 19:18:32.050954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.050655 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 19:18:32.051112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.050647 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 19:18:32.054497 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.054473 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79d7459bb-tfr5x"] Sep 10 19:18:32.074115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.074064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:32.143122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.143087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-metrics-certs\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.143301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.143127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhf6\" (UniqueName: \"kubernetes.io/projected/50470656-22df-414c-8263-43e9e19439ae-kube-api-access-pkhf6\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.143301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.143156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-webhook-cert\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.146056 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.146020 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:32.190045 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.190011 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55cd595499-pvrml"] Sep 10 19:18:32.193488 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.193466 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.208362 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.208339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cd595499-pvrml"] Sep 10 19:18:32.243758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.243668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-webhook-cert\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.243758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.243725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkbgn\" (UniqueName: \"kubernetes.io/projected/e8d7b5ef-65de-4f58-92da-f609ca9daff5-kube-api-access-kkbgn\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.243981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.243770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-metrics-certs\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.243981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.243796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhf6\" (UniqueName: \"kubernetes.io/projected/50470656-22df-414c-8263-43e9e19439ae-kube-api-access-pkhf6\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.243981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.243841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-metrics-certs\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.243981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.243863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-webhook-cert\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.246358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.246336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-metrics-certs\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.246490 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.246400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-webhook-cert\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.253728 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.253696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhf6\" (UniqueName: \"kubernetes.io/projected/50470656-22df-414c-8263-43e9e19439ae-kube-api-access-pkhf6\") pod \"maas-controller-79d7459bb-tfr5x\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.344640 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.344607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-metrics-certs\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.344860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.344663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-webhook-cert\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.344860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.344718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkbgn\" (UniqueName: \"kubernetes.io/projected/e8d7b5ef-65de-4f58-92da-f609ca9daff5-kube-api-access-kkbgn\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.347307 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.347277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-webhook-cert\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.347465 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.347428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-metrics-certs\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.353982 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.353956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkbgn\" (UniqueName: \"kubernetes.io/projected/e8d7b5ef-65de-4f58-92da-f609ca9daff5-kube-api-access-kkbgn\") pod \"maas-controller-55cd595499-pvrml\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.358685 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.358662 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:32.503624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.503539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:32.634218 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.634094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79d7459bb-tfr5x"] Sep 10 19:18:32.636874 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:18:32.636841 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50470656_22df_414c_8263_43e9e19439ae.slice/crio-33008c2346aa230bca04ec87494bc74c338bf29ad001371211dea09faf8be326 WatchSource:0}: Error finding container 33008c2346aa230bca04ec87494bc74c338bf29ad001371211dea09faf8be326: Status 404 returned error can't find the container with id 33008c2346aa230bca04ec87494bc74c338bf29ad001371211dea09faf8be326 Sep 10 19:18:32.778288 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.778261 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cd595499-pvrml"] Sep 10 19:18:32.779471 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:18:32.779447 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d7b5ef_65de_4f58_92da_f609ca9daff5.slice/crio-35437ffb9c8174a819d38978b831795cc6380fb550968e685504b7493cda17fa WatchSource:0}: Error finding container 35437ffb9c8174a819d38978b831795cc6380fb550968e685504b7493cda17fa: Status 404 returned error can't find the container with id 35437ffb9c8174a819d38978b831795cc6380fb550968e685504b7493cda17fa Sep 10 19:18:32.930860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.930817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-pvrml" event={"ID":"e8d7b5ef-65de-4f58-92da-f609ca9daff5","Type":"ContainerStarted","Data":"35437ffb9c8174a819d38978b831795cc6380fb550968e685504b7493cda17fa"} Sep 10 19:18:32.931954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:32.931933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-tfr5x" event={"ID":"50470656-22df-414c-8263-43e9e19439ae","Type":"ContainerStarted","Data":"33008c2346aa230bca04ec87494bc74c338bf29ad001371211dea09faf8be326"} Sep 10 19:18:35.391750 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.391725 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 19:18:35.612713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.612617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:35.712970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.712924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:18:35.954016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.953913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-pvrml" event={"ID":"e8d7b5ef-65de-4f58-92da-f609ca9daff5","Type":"ContainerStarted","Data":"0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f"} Sep 10 19:18:35.954016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.954007 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:35.955679 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.955645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-tfr5x" event={"ID":"50470656-22df-414c-8263-43e9e19439ae","Type":"ContainerStarted","Data":"2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc"} Sep 10 19:18:35.955805 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.955703 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:35.986350 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:35.986297 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55cd595499-pvrml" podStartSLOduration=1.371767141 podStartE2EDuration="3.986282643s" totalImagesPullingTime="2.614515502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:18:35.984042187 +0000 UTC m=+481.900698216" watchObservedRunningTime="2026-09-10 19:18:35.986282643 +0000 UTC m=+481.902938671" Sep 10 19:18:36.016416 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:36.016337 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79d7459bb-tfr5x" podStartSLOduration=2.2654049609999998 podStartE2EDuration="5.016315398s" totalImagesPullingTime="2.750910437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:18:36.012458818 +0000 UTC m=+481.929114847" watchObservedRunningTime="2026-09-10 19:18:36.016315398 +0000 UTC m=+481.932971427" Sep 10 19:18:46.963329 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:46.963300 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:46.963797 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:46.963389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:18:47.027016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.026979 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79d7459bb-tfr5x"] Sep 10 19:18:47.027283 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.027253 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79d7459bb-tfr5x" podUID="50470656-22df-414c-8263-43e9e19439ae" containerName="manager" containerID="cri-o://2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc" gracePeriod=30 Sep 10 19:18:47.276211 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.276189 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:47.370670 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.370629 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-metrics-certs\") pod \"50470656-22df-414c-8263-43e9e19439ae\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " Sep 10 19:18:47.370843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.370685 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkhf6\" (UniqueName: \"kubernetes.io/projected/50470656-22df-414c-8263-43e9e19439ae-kube-api-access-pkhf6\") pod \"50470656-22df-414c-8263-43e9e19439ae\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " Sep 10 19:18:47.370843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.370724 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-webhook-cert\") pod \"50470656-22df-414c-8263-43e9e19439ae\" (UID: \"50470656-22df-414c-8263-43e9e19439ae\") " Sep 10 19:18:47.372993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.372958 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "50470656-22df-414c-8263-43e9e19439ae" (UID: "50470656-22df-414c-8263-43e9e19439ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:18:47.373115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.372991 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "50470656-22df-414c-8263-43e9e19439ae" (UID: "50470656-22df-414c-8263-43e9e19439ae"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:18:47.373115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.373010 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50470656-22df-414c-8263-43e9e19439ae-kube-api-access-pkhf6" (OuterVolumeSpecName: "kube-api-access-pkhf6") pod "50470656-22df-414c-8263-43e9e19439ae" (UID: "50470656-22df-414c-8263-43e9e19439ae"). InnerVolumeSpecName "kube-api-access-pkhf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:18:47.471808 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.471770 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-webhook-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:18:47.471808 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.471801 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50470656-22df-414c-8263-43e9e19439ae-metrics-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:18:47.471808 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:47.471811 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkhf6\" (UniqueName: \"kubernetes.io/projected/50470656-22df-414c-8263-43e9e19439ae-kube-api-access-pkhf6\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:18:48.021513 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.021478 2576 generic.go:356] "Generic (PLEG): container finished" podID="50470656-22df-414c-8263-43e9e19439ae" containerID="2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc" exitCode=0 Sep 10 19:18:48.021984 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.021521 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-tfr5x" Sep 10 19:18:48.021984 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.021576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-tfr5x" event={"ID":"50470656-22df-414c-8263-43e9e19439ae","Type":"ContainerDied","Data":"2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc"} Sep 10 19:18:48.021984 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.021625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-tfr5x" event={"ID":"50470656-22df-414c-8263-43e9e19439ae","Type":"ContainerDied","Data":"33008c2346aa230bca04ec87494bc74c338bf29ad001371211dea09faf8be326"} Sep 10 19:18:48.021984 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.021646 2576 scope.go:117] "RemoveContainer" containerID="2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc" Sep 10 19:18:48.031789 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.031583 2576 scope.go:117] "RemoveContainer" containerID="2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc" Sep 10 19:18:48.031898 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:18:48.031880 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc\": container with ID starting with 2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc not found: ID does not exist" containerID="2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc" Sep 10 19:18:48.031939 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.031905 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc"} err="failed to get container status \"2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc\": rpc error: code = NotFound desc = could not find container \"2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc\": container with ID starting with 2a6ec9b6b05f64a8c7ca75f0d7bfff725487a799cf41be9d6a0d6e588fa32dbc not found: ID does not exist" Sep 10 19:18:48.044565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.044540 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79d7459bb-tfr5x"] Sep 10 19:18:48.050818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.050794 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79d7459bb-tfr5x"] Sep 10 19:18:48.601794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:18:48.601761 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50470656-22df-414c-8263-43e9e19439ae" path="/var/lib/kubelet/pods/50470656-22df-414c-8263-43e9e19439ae/volumes" Sep 10 19:19:05.624826 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:05.624784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:05.730388 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:05.730333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:07.035639 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.035605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:07.137218 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.137176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:07.247396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.247343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:07.343758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.343668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:07.688089 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.687984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:07.859630 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.859598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:07.923178 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.923148 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd"] Sep 10 19:19:07.923555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.923538 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50470656-22df-414c-8263-43e9e19439ae" containerName="manager" Sep 10 19:19:07.923626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.923559 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50470656-22df-414c-8263-43e9e19439ae" containerName="manager" Sep 10 19:19:07.923673 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.923629 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50470656-22df-414c-8263-43e9e19439ae" containerName="manager" Sep 10 19:19:07.926596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.926580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:07.929276 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.929255 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 19:19:07.942297 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.942240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd"] Sep 10 19:19:07.952517 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.952489 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fbqwq"] Sep 10 19:19:07.955560 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.955545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:07.958261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.958245 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gskvn\"" Sep 10 19:19:07.968342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:07.968318 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fbqwq"] Sep 10 19:19:08.042996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.042962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4nd\" (UniqueName: \"kubernetes.io/projected/59022858-10e9-425b-9c9a-dc023040fe85-kube-api-access-jm4nd\") pod \"payload-pre-processing-59fbc47bf8-g92sd\" (UID: \"59022858-10e9-425b-9c9a-dc023040fe85\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:08.043454 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.043056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78633393-0b05-45f7-925c-2c41aceca7b4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fbqwq\" (UID: \"78633393-0b05-45f7-925c-2c41aceca7b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:08.043454 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.043129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59022858-10e9-425b-9c9a-dc023040fe85-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-g92sd\" (UID: \"59022858-10e9-425b-9c9a-dc023040fe85\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:08.043454 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.043156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2w2q\" (UniqueName: \"kubernetes.io/projected/78633393-0b05-45f7-925c-2c41aceca7b4-kube-api-access-b2w2q\") pod \"payload-processing-5b54b7fbb7-fbqwq\" (UID: \"78633393-0b05-45f7-925c-2c41aceca7b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:08.144206 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.144167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4nd\" (UniqueName: \"kubernetes.io/projected/59022858-10e9-425b-9c9a-dc023040fe85-kube-api-access-jm4nd\") pod \"payload-pre-processing-59fbc47bf8-g92sd\" (UID: \"59022858-10e9-425b-9c9a-dc023040fe85\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:08.144395 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.144228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78633393-0b05-45f7-925c-2c41aceca7b4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fbqwq\" (UID: \"78633393-0b05-45f7-925c-2c41aceca7b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:08.144395 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.144249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59022858-10e9-425b-9c9a-dc023040fe85-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-g92sd\" (UID: \"59022858-10e9-425b-9c9a-dc023040fe85\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:08.144395 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.144269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2w2q\" (UniqueName: \"kubernetes.io/projected/78633393-0b05-45f7-925c-2c41aceca7b4-kube-api-access-b2w2q\") pod \"payload-processing-5b54b7fbb7-fbqwq\" (UID: \"78633393-0b05-45f7-925c-2c41aceca7b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:08.145090 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.145071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59022858-10e9-425b-9c9a-dc023040fe85-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-g92sd\" (UID: \"59022858-10e9-425b-9c9a-dc023040fe85\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:08.145203 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.145176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78633393-0b05-45f7-925c-2c41aceca7b4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fbqwq\" (UID: \"78633393-0b05-45f7-925c-2c41aceca7b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:08.154095 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.154066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4nd\" (UniqueName: \"kubernetes.io/projected/59022858-10e9-425b-9c9a-dc023040fe85-kube-api-access-jm4nd\") pod \"payload-pre-processing-59fbc47bf8-g92sd\" (UID: \"59022858-10e9-425b-9c9a-dc023040fe85\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:08.154416 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.154365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2w2q\" (UniqueName: \"kubernetes.io/projected/78633393-0b05-45f7-925c-2c41aceca7b4-kube-api-access-b2w2q\") pod \"payload-processing-5b54b7fbb7-fbqwq\" (UID: \"78633393-0b05-45f7-925c-2c41aceca7b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:08.237176 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.237081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:08.266115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.266089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:08.534182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.534145 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd"] Sep 10 19:19:08.535264 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:19:08.535235 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59022858_10e9_425b_9c9a_dc023040fe85.slice/crio-66e94822ee76cdbac1fbbc0dddf2686df9736b4d9687ab2964e38b3ea90e3928 WatchSource:0}: Error finding container 66e94822ee76cdbac1fbbc0dddf2686df9736b4d9687ab2964e38b3ea90e3928: Status 404 returned error can't find the container with id 66e94822ee76cdbac1fbbc0dddf2686df9736b4d9687ab2964e38b3ea90e3928 Sep 10 19:19:08.555427 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:08.555405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fbqwq"] Sep 10 19:19:08.560393 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:19:08.560345 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78633393_0b05_45f7_925c_2c41aceca7b4.slice/crio-b5bb7875dd2a58d7f98bbdaab3f351be77dc2288e8b89aabed57c932df4bc9a4 WatchSource:0}: Error finding container b5bb7875dd2a58d7f98bbdaab3f351be77dc2288e8b89aabed57c932df4bc9a4: Status 404 returned error can't find the container with id b5bb7875dd2a58d7f98bbdaab3f351be77dc2288e8b89aabed57c932df4bc9a4 Sep 10 19:19:09.133530 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:09.133491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" event={"ID":"78633393-0b05-45f7-925c-2c41aceca7b4","Type":"ContainerStarted","Data":"b5bb7875dd2a58d7f98bbdaab3f351be77dc2288e8b89aabed57c932df4bc9a4"} Sep 10 19:19:09.134876 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:09.134847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" event={"ID":"59022858-10e9-425b-9c9a-dc023040fe85","Type":"ContainerStarted","Data":"66e94822ee76cdbac1fbbc0dddf2686df9736b4d9687ab2964e38b3ea90e3928"} Sep 10 19:19:11.149118 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:11.149077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" event={"ID":"78633393-0b05-45f7-925c-2c41aceca7b4","Type":"ContainerStarted","Data":"9c2f42e2fee87a85f5461c681124b773b5342d7743b070614ab8dc001bc4706a"} Sep 10 19:19:11.149631 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:11.149179 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:11.150848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:11.150823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" event={"ID":"59022858-10e9-425b-9c9a-dc023040fe85","Type":"ContainerStarted","Data":"aead414885c9707bebe3b70a8012170dcaf942bbd43e1d3e3efa65237f56d1e3"} Sep 10 19:19:11.150987 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:11.150897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:11.167874 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:11.167825 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" podStartSLOduration=2.254905515 podStartE2EDuration="4.167810356s" totalImagesPullingTime="1.912904841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:19:11.165566871 +0000 UTC m=+517.082222902" watchObservedRunningTime="2026-09-10 19:19:11.167810356 +0000 UTC m=+517.084466384" Sep 10 19:19:11.182575 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:11.182528 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" podStartSLOduration=2.251177676 podStartE2EDuration="4.182513352s" totalImagesPullingTime="1.931335676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:19:11.181869459 +0000 UTC m=+517.098525488" watchObservedRunningTime="2026-09-10 19:19:11.182513352 +0000 UTC m=+517.099169381" Sep 10 19:19:18.419473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:18.419431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:18.489486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:18.489456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:22.157558 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:22.157527 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g92sd" Sep 10 19:19:22.158014 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:22.157903 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fbqwq" Sep 10 19:19:24.484169 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:24.484134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:35.843311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:35.843274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:45.715755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.715707 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf"] Sep 10 19:19:45.722477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.722453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.725270 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.725247 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xvnhg\"" Sep 10 19:19:45.725429 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.725388 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 19:19:45.725429 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.725386 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 10 19:19:45.726166 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.726152 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 19:19:45.730966 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.730895 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf"] Sep 10 19:19:45.751038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.751001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:19:45.851714 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.851681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.851714 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.851715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlzb\" (UniqueName: \"kubernetes.io/projected/8cec2d25-d6ba-4b85-8c37-967156c18598-kube-api-access-bdlzb\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.851938 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.851743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.851938 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.851774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cec2d25-d6ba-4b85-8c37-967156c18598-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.851938 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.851843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.851938 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.851890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953230 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953452 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlzb\" (UniqueName: \"kubernetes.io/projected/8cec2d25-d6ba-4b85-8c37-967156c18598-kube-api-access-bdlzb\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953452 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cec2d25-d6ba-4b85-8c37-967156c18598-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953751 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953751 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.953893 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.953872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.955949 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.955930 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cec2d25-d6ba-4b85-8c37-967156c18598-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.956213 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.956193 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cec2d25-d6ba-4b85-8c37-967156c18598-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:45.961757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:45.961732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlzb\" (UniqueName: \"kubernetes.io/projected/8cec2d25-d6ba-4b85-8c37-967156c18598-kube-api-access-bdlzb\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf\" (UID: \"8cec2d25-d6ba-4b85-8c37-967156c18598\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:46.035408 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:46.035299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:46.312130 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:19:46.312098 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cec2d25_d6ba_4b85_8c37_967156c18598.slice/crio-35c1e9f1ce98c87d711c11bfbdc64503f7b30cd30ec26cb01e6c64bea2be9d22 WatchSource:0}: Error finding container 35c1e9f1ce98c87d711c11bfbdc64503f7b30cd30ec26cb01e6c64bea2be9d22: Status 404 returned error can't find the container with id 35c1e9f1ce98c87d711c11bfbdc64503f7b30cd30ec26cb01e6c64bea2be9d22 Sep 10 19:19:46.312424 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:46.312400 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf"] Sep 10 19:19:46.330214 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:46.330184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" event={"ID":"8cec2d25-d6ba-4b85-8c37-967156c18598","Type":"ContainerStarted","Data":"35c1e9f1ce98c87d711c11bfbdc64503f7b30cd30ec26cb01e6c64bea2be9d22"} Sep 10 19:19:48.343647 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:48.343604 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" event={"ID":"8cec2d25-d6ba-4b85-8c37-967156c18598","Type":"ContainerStarted","Data":"cc9ca602c36cf61c157dfde8e728d765de50af240c9d92145724b0932872f4ca"} Sep 10 19:19:48.364942 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:48.364877 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" podStartSLOduration=2.327077143 podStartE2EDuration="3.364862378s" totalImagesPullingTime="1.037785235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:19:48.362043331 +0000 UTC m=+554.278699373" watchObservedRunningTime="2026-09-10 19:19:48.364862378 +0000 UTC m=+554.281518406" Sep 10 19:19:56.036414 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:56.036357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:56.036808 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:56.036456 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:56.050415 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:56.050091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:19:56.397060 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:19:56.396973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf" Sep 10 19:20:02.615978 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:02.615947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:20:06.219122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.219092 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm"] Sep 10 19:20:06.222347 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.222331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.224809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.224787 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 19:20:06.233507 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.233486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm"] Sep 10 19:20:06.326272 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.326244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:20:06.334138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.334115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.334217 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.334172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6887099-3834-46d4-81c5-d8e83ab7e4bc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.334261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.334242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.334298 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.334268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8gx\" (UniqueName: \"kubernetes.io/projected/d6887099-3834-46d4-81c5-d8e83ab7e4bc-kube-api-access-2p8gx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.334345 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.334333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.334408 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.334357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.435016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.434976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.435016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.435011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8gx\" (UniqueName: \"kubernetes.io/projected/d6887099-3834-46d4-81c5-d8e83ab7e4bc-kube-api-access-2p8gx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.435259 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.435056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.435259 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.435074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.435259 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.435109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.435985 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.435901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.436133 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.436085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6887099-3834-46d4-81c5-d8e83ab7e4bc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.436301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.436109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.440398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.436491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.440398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.438557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6887099-3834-46d4-81c5-d8e83ab7e4bc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.444712 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.444685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6887099-3834-46d4-81c5-d8e83ab7e4bc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.445904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.445883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8gx\" (UniqueName: \"kubernetes.io/projected/d6887099-3834-46d4-81c5-d8e83ab7e4bc-kube-api-access-2p8gx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vccm\" (UID: \"d6887099-3834-46d4-81c5-d8e83ab7e4bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.533031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.532940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:06.807047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:06.807018 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm"] Sep 10 19:20:06.808485 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:20:06.808457 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6887099_3834_46d4_81c5_d8e83ab7e4bc.slice/crio-ab808f6d04112dda87fb5ab0ebf51b3ac118aef86ec82fd98e10e133e38ab556 WatchSource:0}: Error finding container ab808f6d04112dda87fb5ab0ebf51b3ac118aef86ec82fd98e10e133e38ab556: Status 404 returned error can't find the container with id ab808f6d04112dda87fb5ab0ebf51b3ac118aef86ec82fd98e10e133e38ab556 Sep 10 19:20:07.449233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:07.449160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" event={"ID":"d6887099-3834-46d4-81c5-d8e83ab7e4bc","Type":"ContainerStarted","Data":"ec269006328c5a89ec3b7109aed5fbeb164a050b57af44b6b8b1ee45a3811122"} Sep 10 19:20:07.449233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:07.449206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" event={"ID":"d6887099-3834-46d4-81c5-d8e83ab7e4bc","Type":"ContainerStarted","Data":"ab808f6d04112dda87fb5ab0ebf51b3ac118aef86ec82fd98e10e133e38ab556"} Sep 10 19:20:07.468580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:07.468523 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" podStartSLOduration=1.26909603 podStartE2EDuration="1.468506208s" totalImagesPullingTime="199.410178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:07.468211194 +0000 UTC m=+573.384867236" watchObservedRunningTime="2026-09-10 19:20:07.468506208 +0000 UTC m=+573.385162237" Sep 10 19:20:08.553100 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:08.553064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:20:16.533338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:16.533301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:16.533758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:16.533396 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:16.545889 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:16.545858 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:17.172033 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:17.171997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:20:17.514401 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:17.514308 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vccm" Sep 10 19:20:21.435502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:21.435468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:20:21.538923 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:21.538893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:20:34.548198 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:34.548169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:20:34.548707 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:34.548486 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:20:34.860029 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:34.859956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:20:50.748906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:20:50.748865 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:10.706506 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:10.706476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:10.807650 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:10.807618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:42.771406 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:42.771352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:42.888442 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:42.888405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:43.906540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:43.906504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:44.000075 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:44.000039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:59.773943 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:59.773911 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:21:59.880443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:21:59.880415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:00.171434 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:00.171356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:00.519013 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:00.518932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:01.214491 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:01.214460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:01.308904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:01.308870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:01.677860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:01.677829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:03.142976 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:03.142940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:03.237561 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:03.237526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:04.538279 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:04.538251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:04.694586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:04.694557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:04.904060 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:04.903986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:05.061139 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.061112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:05.153199 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.153173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:05.174096 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.174011 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6"] Sep 10 19:22:05.177444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.177412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.179941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.179914 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6b562c\"" Sep 10 19:22:05.189025 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.189001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6"] Sep 10 19:22:05.234972 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.234940 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn"] Sep 10 19:22:05.238297 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.238279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.240753 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.240732 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6b562c-dockercfg-dwlkk\"" Sep 10 19:22:05.249269 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.249249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn"] Sep 10 19:22:05.262363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.262327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzhb\" (UniqueName: \"kubernetes.io/projected/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-kube-api-access-swzhb\") pod \"payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.262512 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.262404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmg5s\" (UniqueName: \"kubernetes.io/projected/6917793d-c854-45ac-8ccf-8e837fc97110-kube-api-access-mmg5s\") pod \"payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.262512 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.262484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.262596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.262548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6917793d-c854-45ac-8ccf-8e837fc97110-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.363736 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.363694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.363905 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.363785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6917793d-c854-45ac-8ccf-8e837fc97110-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.363905 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.363843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzhb\" (UniqueName: \"kubernetes.io/projected/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-kube-api-access-swzhb\") pod \"payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.363905 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.363882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmg5s\" (UniqueName: \"kubernetes.io/projected/6917793d-c854-45ac-8ccf-8e837fc97110-kube-api-access-mmg5s\") pod \"payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.364513 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.364493 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.364770 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.364741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6917793d-c854-45ac-8ccf-8e837fc97110-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.374420 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.374392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmg5s\" (UniqueName: \"kubernetes.io/projected/6917793d-c854-45ac-8ccf-8e837fc97110-kube-api-access-mmg5s\") pod \"payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.374790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.374769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzhb\" (UniqueName: \"kubernetes.io/projected/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-kube-api-access-swzhb\") pod \"payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.488490 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.488415 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:05.549610 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.549529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:05.764831 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.764809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6"] Sep 10 19:22:05.766130 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:05.766095 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6917793d_c854_45ac_8ccf_8e837fc97110.slice/crio-fd92b4bec1a82409fb26640f16b67fee14aecb2b0d2243c7a4aadd25cefbea9d WatchSource:0}: Error finding container fd92b4bec1a82409fb26640f16b67fee14aecb2b0d2243c7a4aadd25cefbea9d: Status 404 returned error can't find the container with id fd92b4bec1a82409fb26640f16b67fee14aecb2b0d2243c7a4aadd25cefbea9d Sep 10 19:22:05.830817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:05.830791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn"] Sep 10 19:22:05.831894 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:05.831861 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b1b9cc_8ca7_449d_be53_c0fff0d51a22.slice/crio-a08dd8bf5cfa39f1ef31a7478b778111ff536c6aaabbeb752179f202d414518d WatchSource:0}: Error finding container a08dd8bf5cfa39f1ef31a7478b778111ff536c6aaabbeb752179f202d414518d: Status 404 returned error can't find the container with id a08dd8bf5cfa39f1ef31a7478b778111ff536c6aaabbeb752179f202d414518d Sep 10 19:22:06.076125 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.076084 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" event={"ID":"82b1b9cc-8ca7-449d-be53-c0fff0d51a22","Type":"ContainerStarted","Data":"85e3de1d82c8e83a0afa6067cb61195d8234a2625d9a6a4a717e84a988a79bc7"} Sep 10 19:22:06.076318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.076178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" event={"ID":"82b1b9cc-8ca7-449d-be53-c0fff0d51a22","Type":"ContainerStarted","Data":"a08dd8bf5cfa39f1ef31a7478b778111ff536c6aaabbeb752179f202d414518d"} Sep 10 19:22:06.076318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.076227 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:06.078282 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.078094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" event={"ID":"6917793d-c854-45ac-8ccf-8e837fc97110","Type":"ContainerStarted","Data":"d490621ab7b24eb076ebc24a0339a9712998e56eaba272af4b42381353268c04"} Sep 10 19:22:06.078282 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.078161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" event={"ID":"6917793d-c854-45ac-8ccf-8e837fc97110","Type":"ContainerStarted","Data":"fd92b4bec1a82409fb26640f16b67fee14aecb2b0d2243c7a4aadd25cefbea9d"} Sep 10 19:22:06.078282 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.078206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:06.094917 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.094867 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" podStartSLOduration=1.094850038 podStartE2EDuration="1.094850038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:06.09213787 +0000 UTC m=+692.008793901" watchObservedRunningTime="2026-09-10 19:22:06.094850038 +0000 UTC m=+692.011506069" Sep 10 19:22:06.108967 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:06.108910 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" podStartSLOduration=1.108893418 podStartE2EDuration="1.108893418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:06.108080792 +0000 UTC m=+692.024736821" watchObservedRunningTime="2026-09-10 19:22:06.108893418 +0000 UTC m=+692.025549447" Sep 10 19:22:07.637752 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:07.637719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:07.738274 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:07.738242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:07.867735 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:07.867702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:08.149004 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.148970 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq"] Sep 10 19:22:08.152750 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.152726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.155621 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.155588 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8db50f5a-openshift-default-dockercfg-j9zmr\"" Sep 10 19:22:08.168732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.168704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq"] Sep 10 19:22:08.183418 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.183389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:08.191628 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-certs\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191729 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-socket\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191729 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqvm\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-kube-api-access-fhqvm\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191729 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-credential-socket\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191729 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d93171-bc91-43a9-890c-9eaadb36ee50-istiod-ca-cert\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191863 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-envoy\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191863 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-data\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191863 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-token\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.191863 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.191858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-podinfo\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.292910 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.292877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-credential-socket\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.292919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d93171-bc91-43a9-890c-9eaadb36ee50-istiod-ca-cert\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-envoy\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-data\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-token\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293257 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-podinfo\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293257 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-certs\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293351 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-socket\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293351 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqvm\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-kube-api-access-fhqvm\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293567 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-data\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293651 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-certs\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293651 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-credential-socket\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293708 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d93171-bc91-43a9-890c-9eaadb36ee50-istiod-ca-cert\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.293864 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.293842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-socket\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.295693 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.295659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-envoy\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.295861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.295842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-podinfo\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.301805 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.301781 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-token\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.301805 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.301788 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqvm\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-kube-api-access-fhqvm\") pod \"e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.358526 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.358496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:08.458890 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.458800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:08.465210 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.465190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:08.758835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.758807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq"] Sep 10 19:22:08.760143 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:08.760110 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d93171_bc91_43a9_890c_9eaadb36ee50.slice/crio-a88bf12d05e50c24d14b9b9b010c5b36ca88966bfd88729a07b26a51b9ba9531 WatchSource:0}: Error finding container a88bf12d05e50c24d14b9b9b010c5b36ca88966bfd88729a07b26a51b9ba9531: Status 404 returned error can't find the container with id a88bf12d05e50c24d14b9b9b010c5b36ca88966bfd88729a07b26a51b9ba9531 Sep 10 19:22:08.762883 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.762845 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:08.762964 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.762930 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:08.763012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.762972 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:08.789562 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:08.789535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:09.102248 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:09.102214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" event={"ID":"58d93171-bc91-43a9-890c-9eaadb36ee50","Type":"ContainerStarted","Data":"3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57"} Sep 10 19:22:09.102444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:09.102298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" event={"ID":"58d93171-bc91-43a9-890c-9eaadb36ee50","Type":"ContainerStarted","Data":"a88bf12d05e50c24d14b9b9b010c5b36ca88966bfd88729a07b26a51b9ba9531"} Sep 10 19:22:09.125068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:09.125020 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" podStartSLOduration=1.1250031 podStartE2EDuration="1.1250031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:09.122288264 +0000 UTC m=+695.038944317" watchObservedRunningTime="2026-09-10 19:22:09.1250031 +0000 UTC m=+695.041659129" Sep 10 19:22:09.359226 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:09.359145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:09.465928 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:09.465893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:09.470602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:09.470575 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:10.107620 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:10.107585 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:10.108502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:10.108480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:10.558577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:10.558548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:10.695675 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:10.695644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:11.861503 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:11.861458 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:11.967426 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:11.967392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:12.053416 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.053365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:12.323317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.323266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:12.455293 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.455248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:12.582855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.582778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:12.684862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.684835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:12.756597 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.754749 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz"] Sep 10 19:22:12.759059 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.759033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:12.762839 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.762796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8db50f5a-dockercfg-bmbzm\"" Sep 10 19:22:12.765021 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.764717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8db50f5a\"" Sep 10 19:22:12.770112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.770089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz"] Sep 10 19:22:12.837109 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.837029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-plugins-config-volume\") pod \"payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:12.837109 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.837062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbvm\" (UniqueName: \"kubernetes.io/projected/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-kube-api-access-4wbvm\") pod \"payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:12.938543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.938507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-plugins-config-volume\") pod \"payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:12.938543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.938542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbvm\" (UniqueName: \"kubernetes.io/projected/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-kube-api-access-4wbvm\") pod \"payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:12.939360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.939337 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-plugins-config-volume\") pod \"payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:12.948110 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:12.948073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbvm\" (UniqueName: \"kubernetes.io/projected/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-kube-api-access-4wbvm\") pod \"payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:13.079420 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:13.079356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:13.358155 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:13.358130 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz"] Sep 10 19:22:13.360256 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:13.360230 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b71b45_ca5c_41b6_b36c_95ea623d5ce5.slice/crio-6511a4d296a176b358372a3fabbfe00efe6c96521edc8d366cc6d965f940ac4d WatchSource:0}: Error finding container 6511a4d296a176b358372a3fabbfe00efe6c96521edc8d366cc6d965f940ac4d: Status 404 returned error can't find the container with id 6511a4d296a176b358372a3fabbfe00efe6c96521edc8d366cc6d965f940ac4d Sep 10 19:22:13.732002 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:13.731899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:13.830142 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:13.830110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:14.131999 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.131953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" event={"ID":"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5","Type":"ContainerStarted","Data":"07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe"} Sep 10 19:22:14.132536 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.132047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" event={"ID":"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5","Type":"ContainerStarted","Data":"6511a4d296a176b358372a3fabbfe00efe6c96521edc8d366cc6d965f940ac4d"} Sep 10 19:22:14.132536 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.132070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:14.153326 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.153268 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" podStartSLOduration=2.153253008 podStartE2EDuration="2.153253008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:14.150573069 +0000 UTC m=+700.067229100" watchObservedRunningTime="2026-09-10 19:22:14.153253008 +0000 UTC m=+700.069909037" Sep 10 19:22:14.246041 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.245994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:14.345598 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.345566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:14.369056 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.369027 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp"] Sep 10 19:22:14.372484 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.372462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.375296 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.375276 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3c7a5e-dockercfg-nnphb\"" Sep 10 19:22:14.375414 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.375286 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3c7a5e\"" Sep 10 19:22:14.385944 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.385886 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp"] Sep 10 19:22:14.461100 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.461032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db16422-fc7a-46a5-bed9-7a2b3aa34868-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.461362 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.461183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72jh\" (UniqueName: \"kubernetes.io/projected/7db16422-fc7a-46a5-bed9-7a2b3aa34868-kube-api-access-x72jh\") pod \"payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.562193 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.562160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72jh\" (UniqueName: \"kubernetes.io/projected/7db16422-fc7a-46a5-bed9-7a2b3aa34868-kube-api-access-x72jh\") pod \"payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.562357 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.562245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db16422-fc7a-46a5-bed9-7a2b3aa34868-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.562972 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.562953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db16422-fc7a-46a5-bed9-7a2b3aa34868-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.571831 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.571806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72jh\" (UniqueName: \"kubernetes.io/projected/7db16422-fc7a-46a5-bed9-7a2b3aa34868-kube-api-access-x72jh\") pod \"payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.684539 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.684453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:14.742114 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.741904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:14.959644 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:14.959613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp"] Sep 10 19:22:14.960100 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:14.960074 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db16422_fc7a_46a5_bed9_7a2b3aa34868.slice/crio-9cd52894219d127a6672e843c20f33dfcc5ee8a80fdec0afc7b1d8bee251b703 WatchSource:0}: Error finding container 9cd52894219d127a6672e843c20f33dfcc5ee8a80fdec0afc7b1d8bee251b703: Status 404 returned error can't find the container with id 9cd52894219d127a6672e843c20f33dfcc5ee8a80fdec0afc7b1d8bee251b703 Sep 10 19:22:15.140803 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:15.140766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" event={"ID":"7db16422-fc7a-46a5-bed9-7a2b3aa34868","Type":"ContainerStarted","Data":"ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29"} Sep 10 19:22:15.141283 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:15.140829 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" event={"ID":"7db16422-fc7a-46a5-bed9-7a2b3aa34868","Type":"ContainerStarted","Data":"9cd52894219d127a6672e843c20f33dfcc5ee8a80fdec0afc7b1d8bee251b703"} Sep 10 19:22:15.141283 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:15.140991 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:15.159361 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:15.159309 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" podStartSLOduration=1.159293881 podStartE2EDuration="1.159293881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:15.157892601 +0000 UTC m=+701.074548630" watchObservedRunningTime="2026-09-10 19:22:15.159293881 +0000 UTC m=+701.075949909" Sep 10 19:22:17.085147 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:17.085111 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:22:17.085655 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:17.085237 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:22:18.552752 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:18.552709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:18.739267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:18.739219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:18.833765 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:18.833689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:18.918953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:18.918912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:21.587954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:21.587914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:24.050330 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:24.050281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:25.133520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:25.133485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:25.144634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:25.144605 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:25.251982 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:25.251937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:25.380804 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:25.380768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:26.150645 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:26.150564 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:22:26.853061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:26.853032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:28.077662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.077626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:28.418269 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.418189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:28.720115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.720036 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt"] Sep 10 19:22:28.723625 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.723604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.727237 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.727216 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b960a2-openshift-default-dockercfg-wvjs2\"" Sep 10 19:22:28.742168 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.742141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt"] Sep 10 19:22:28.760865 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.760834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:28.898540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-podinfo\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtj9\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-kube-api-access-8xtj9\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5b1544-0e93-4aba-b06c-900c311b3db8-istiod-ca-cert\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-data\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-token\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-credential-socket\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-socket\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-envoy\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.898807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.898771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-certs\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.962928 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.962892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:28.999341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-certs\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-podinfo\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtj9\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-kube-api-access-8xtj9\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5b1544-0e93-4aba-b06c-900c311b3db8-istiod-ca-cert\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-data\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-token\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-credential-socket\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-socket\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-envoy\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:28.999847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-certs\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.000069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-data\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.000069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:28.999998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-socket\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.000311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.000287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-credential-socket\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.000697 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.000421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5b1544-0e93-4aba-b06c-900c311b3db8-istiod-ca-cert\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.002015 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.001993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-envoy\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.002190 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.002168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-podinfo\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.014455 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.014425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-token\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.014674 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.014652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtj9\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-kube-api-access-8xtj9\") pod \"e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.035341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.035310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:29.325609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.325576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt"] Sep 10 19:22:29.331916 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:29.331834 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5b1544_0e93_4aba_b06c_900c311b3db8.slice/crio-0a75a14b83cd4e001877043cda1cf76167c8d32cacc610299eb24c46e23f9696 WatchSource:0}: Error finding container 0a75a14b83cd4e001877043cda1cf76167c8d32cacc610299eb24c46e23f9696: Status 404 returned error can't find the container with id 0a75a14b83cd4e001877043cda1cf76167c8d32cacc610299eb24c46e23f9696 Sep 10 19:22:29.333765 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.333729 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:29.333856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.333803 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:29.333856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:29.333831 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:30.227575 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.227544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" event={"ID":"6f5b1544-0e93-4aba-b06c-900c311b3db8","Type":"ContainerStarted","Data":"1e6789eec89b9e6e05995573052425ba051f79c9f8760a2c9a6298383dedf21b"} Sep 10 19:22:30.227819 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.227607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" event={"ID":"6f5b1544-0e93-4aba-b06c-900c311b3db8","Type":"ContainerStarted","Data":"0a75a14b83cd4e001877043cda1cf76167c8d32cacc610299eb24c46e23f9696"} Sep 10 19:22:30.229265 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.229239 2576 generic.go:356] "Generic (PLEG): container finished" podID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerID="3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57" exitCode=0 Sep 10 19:22:30.229367 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.229301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" event={"ID":"58d93171-bc91-43a9-890c-9eaadb36ee50","Type":"ContainerDied","Data":"3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57"} Sep 10 19:22:30.229648 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.229635 2576 scope.go:117] "RemoveContainer" containerID="3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57" Sep 10 19:22:30.230626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.230595 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:30.230738 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.230678 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:30.230738 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.230727 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:30.254785 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:30.254731 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podStartSLOduration=2.254716576 podStartE2EDuration="2.254716576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:30.251432502 +0000 UTC m=+716.168088535" watchObservedRunningTime="2026-09-10 19:22:30.254716576 +0000 UTC m=+716.171372638" Sep 10 19:22:31.038050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.038005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:31.047981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.047947 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:31.245455 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.245410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" event={"ID":"58d93171-bc91-43a9-890c-9eaadb36ee50","Type":"ContainerStarted","Data":"ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a"} Sep 10 19:22:31.245634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.245486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:31.248075 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.248043 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:31.466269 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.466180 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:31.466468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.466431 2576 patch_prober.go:28] interesting pod/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 19:22:31.466556 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:31.466490 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 19:22:32.144759 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:32.144680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:32.466347 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:32.466272 2576 patch_prober.go:28] interesting pod/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 19:22:32.466347 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:32.466327 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 19:22:32.534126 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:32.534086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:32.722483 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:32.722365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:33.056722 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:33.056685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:33.465946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:33.465866 2576 patch_prober.go:28] interesting pod/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 19:22:33.465946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:33.465919 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 19:22:34.184941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.184900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:34.282996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.282962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:34.466104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.466026 2576 patch_prober.go:28] interesting pod/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 19:22:34.466572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.466542 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 19:22:34.546043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.546008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:34.742415 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.742257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:34.848823 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.848783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:34.879505 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.879469 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt"] Sep 10 19:22:34.884609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.884589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:34.887724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.887607 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b960a2\"" Sep 10 19:22:34.887724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.887634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b960a2-dockercfg-z45rb\"" Sep 10 19:22:34.898268 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:34.898244 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt"] Sep 10 19:22:35.065161 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.065123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdkt\" (UniqueName: \"kubernetes.io/projected/4e5f41c6-4458-492d-8943-2dff54201505-kube-api-access-dkdkt\") pod \"payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:35.065347 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.065273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f41c6-4458-492d-8943-2dff54201505-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:35.165807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.165778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f41c6-4458-492d-8943-2dff54201505-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:35.166103 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.166090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdkt\" (UniqueName: \"kubernetes.io/projected/4e5f41c6-4458-492d-8943-2dff54201505-kube-api-access-dkdkt\") pod \"payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:35.166613 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.166591 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f41c6-4458-492d-8943-2dff54201505-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:35.178758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.178729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdkt\" (UniqueName: \"kubernetes.io/projected/4e5f41c6-4458-492d-8943-2dff54201505-kube-api-access-dkdkt\") pod \"payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:35.187835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.187791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:35.196726 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.196704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:35.466107 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.466005 2576 patch_prober.go:28] interesting pod/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 19:22:35.466107 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.466079 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 19:22:35.511316 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:35.508219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt"] Sep 10 19:22:36.074174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.074139 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8"] Sep 10 19:22:36.077798 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.077777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" Sep 10 19:22:36.080580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.080562 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:22:36.081076 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.081057 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:22:36.088354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.088333 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8"] Sep 10 19:22:36.174885 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.174853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb9e892-b079-46c3-a893-e89c996a59d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8\" (UID: \"deb9e892-b079-46c3-a893-e89c996a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" Sep 10 19:22:36.191767 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.191732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:36.275602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.275569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb9e892-b079-46c3-a893-e89c996a59d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8\" (UID: \"deb9e892-b079-46c3-a893-e89c996a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" Sep 10 19:22:36.276496 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.276473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb9e892-b079-46c3-a893-e89c996a59d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8\" (UID: \"deb9e892-b079-46c3-a893-e89c996a59d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" Sep 10 19:22:36.283571 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.283537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" event={"ID":"4e5f41c6-4458-492d-8943-2dff54201505","Type":"ContainerStarted","Data":"71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca"} Sep 10 19:22:36.283746 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.283609 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:36.283746 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.283637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" event={"ID":"4e5f41c6-4458-492d-8943-2dff54201505","Type":"ContainerStarted","Data":"b7720296a54723e7573fcfc5844dbd5df667570ea709f0d9d575e02b259c5713"} Sep 10 19:22:36.305052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.305000 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" podStartSLOduration=2.304978792 podStartE2EDuration="2.304978792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:36.303929455 +0000 UTC m=+722.220585486" watchObservedRunningTime="2026-09-10 19:22:36.304978792 +0000 UTC m=+722.221634821" Sep 10 19:22:36.388565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.388459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" Sep 10 19:22:36.527061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.526934 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:36.528030 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.527838 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:36.529690 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.529669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:36.684007 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.683984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8"] Sep 10 19:22:36.686031 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:36.685987 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb9e892_b079_46c3_a893_e89c996a59d6.slice/crio-9e10fbe0691f69dd0603c4babbffeeac358084df997b2ab3fde08dfdf9a12dd1 WatchSource:0}: Error finding container 9e10fbe0691f69dd0603c4babbffeeac358084df997b2ab3fde08dfdf9a12dd1: Status 404 returned error can't find the container with id 9e10fbe0691f69dd0603c4babbffeeac358084df997b2ab3fde08dfdf9a12dd1 Sep 10 19:22:36.688115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:36.688097 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:22:37.031679 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:37.031572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:37.290006 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:37.289969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" event={"ID":"deb9e892-b079-46c3-a893-e89c996a59d6","Type":"ContainerStarted","Data":"9e10fbe0691f69dd0603c4babbffeeac358084df997b2ab3fde08dfdf9a12dd1"} Sep 10 19:22:37.904249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:37.904220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:39.242761 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:39.242720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:40.237021 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:40.236967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:40.314591 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:40.314554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" event={"ID":"deb9e892-b079-46c3-a893-e89c996a59d6","Type":"ContainerStarted","Data":"3cb322fefe667afaec751e4a4070fec294ac1e131f746d75e2dba1a0f0714de9"} Sep 10 19:22:40.320165 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:40.320135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:40.337502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:40.337432 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" podStartSLOduration=1.610785957 podStartE2EDuration="4.337396282s" totalImagesPullingTime="2.726610325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:40.332809462 +0000 UTC m=+726.249465496" watchObservedRunningTime="2026-09-10 19:22:40.337396282 +0000 UTC m=+726.254052307" Sep 10 19:22:41.323359 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:41.323323 2576 generic.go:356] "Generic (PLEG): container finished" podID="deb9e892-b079-46c3-a893-e89c996a59d6" containerID="3cb322fefe667afaec751e4a4070fec294ac1e131f746d75e2dba1a0f0714de9" exitCode=0 Sep 10 19:22:41.323771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:41.323409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" event={"ID":"deb9e892-b079-46c3-a893-e89c996a59d6","Type":"ContainerDied","Data":"3cb322fefe667afaec751e4a4070fec294ac1e131f746d75e2dba1a0f0714de9"} Sep 10 19:22:42.516191 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:42.516166 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" Sep 10 19:22:42.551077 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:42.551043 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb9e892-b079-46c3-a893-e89c996a59d6-maas-api-service-ca\") pod \"deb9e892-b079-46c3-a893-e89c996a59d6\" (UID: \"deb9e892-b079-46c3-a893-e89c996a59d6\") " Sep 10 19:22:42.551968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:42.551780 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb9e892-b079-46c3-a893-e89c996a59d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "deb9e892-b079-46c3-a893-e89c996a59d6" (UID: "deb9e892-b079-46c3-a893-e89c996a59d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:22:42.552553 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:42.552532 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb9e892-b079-46c3-a893-e89c996a59d6-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:42.571781 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:42.571743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:43.360278 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:43.360240 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" Sep 10 19:22:43.360278 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:43.360257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8" event={"ID":"deb9e892-b079-46c3-a893-e89c996a59d6","Type":"ContainerDied","Data":"9e10fbe0691f69dd0603c4babbffeeac358084df997b2ab3fde08dfdf9a12dd1"} Sep 10 19:22:43.360597 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:43.360315 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e10fbe0691f69dd0603c4babbffeeac358084df997b2ab3fde08dfdf9a12dd1" Sep 10 19:22:47.287563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:47.287521 2576 patch_prober.go:28] interesting pod/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 19:22:47.288117 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:47.287582 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 19:22:47.292625 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:47.292601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:22:48.393999 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:48.393898 2576 generic.go:356] "Generic (PLEG): container finished" podID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerID="1e6789eec89b9e6e05995573052425ba051f79c9f8760a2c9a6298383dedf21b" exitCode=0 Sep 10 19:22:48.393999 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:48.393948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" event={"ID":"6f5b1544-0e93-4aba-b06c-900c311b3db8","Type":"ContainerDied","Data":"1e6789eec89b9e6e05995573052425ba051f79c9f8760a2c9a6298383dedf21b"} Sep 10 19:22:48.394460 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:48.394347 2576 scope.go:117] "RemoveContainer" containerID="1e6789eec89b9e6e05995573052425ba051f79c9f8760a2c9a6298383dedf21b" Sep 10 19:22:48.395541 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:48.395511 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:48.395624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:48.395579 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:48.395624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:48.395608 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:49.402010 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:49.401971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" event={"ID":"6f5b1544-0e93-4aba-b06c-900c311b3db8","Type":"ContainerStarted","Data":"fd85a53eef4ae019e14c228521087e175c3f70ef0fe0e4ad56ef86c73511b785"} Sep 10 19:22:50.035791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:50.035751 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:50.036023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:50.035992 2576 patch_prober.go:28] interesting pod/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 19:22:50.036105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:50.036059 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 19:22:50.116641 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:50.116605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:51.035987 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:51.035944 2576 patch_prober.go:28] interesting pod/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 19:22:51.036512 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:51.036011 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 19:22:51.941987 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:51.941937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:52.036366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:52.036312 2576 patch_prober.go:28] interesting pod/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 19:22:52.037415 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:52.036965 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 19:22:52.852283 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:52.852245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:52.915842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:52.915804 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz"] Sep 10 19:22:52.916115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:52.916087 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" podUID="a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" containerName="payload-processing" containerID="cri-o://07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe" gracePeriod=30 Sep 10 19:22:53.007562 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.007528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:53.050397 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.047573 2576 patch_prober.go:28] interesting pod/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 19:22:53.050397 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.047649 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 19:22:53.104860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.104779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:53.210230 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.210205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:53.226220 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.225526 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:53.264663 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.264630 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-plugins-config-volume\") pod \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " Sep 10 19:22:53.264850 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.264679 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wbvm\" (UniqueName: \"kubernetes.io/projected/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-kube-api-access-4wbvm\") pod \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\" (UID: \"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5\") " Sep 10 19:22:53.265034 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.265006 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" (UID: "a3b71b45-ca5c-41b6-b36c-95ea623d5ce5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:22:53.265170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.265152 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8"] Sep 10 19:22:53.266999 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.266976 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-kube-api-access-4wbvm" (OuterVolumeSpecName: "kube-api-access-4wbvm") pod "a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" (UID: "a3b71b45-ca5c-41b6-b36c-95ea623d5ce5"). InnerVolumeSpecName "kube-api-access-4wbvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:22:53.282229 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.282204 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8db50f5a-x8rk8"] Sep 10 19:22:53.366654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.366555 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:53.366877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.366859 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbvm\" (UniqueName: \"kubernetes.io/projected/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5-kube-api-access-4wbvm\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:53.398363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.398329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:53.405319 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.405290 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq"] Sep 10 19:22:53.405550 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.405526 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" containerID="cri-o://ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a" gracePeriod=30 Sep 10 19:22:53.427628 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.427596 2576 generic.go:356] "Generic (PLEG): container finished" podID="a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" containerID="07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe" exitCode=0 Sep 10 19:22:53.427756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.427635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" event={"ID":"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5","Type":"ContainerDied","Data":"07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe"} Sep 10 19:22:53.427756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.427642 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" Sep 10 19:22:53.427756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.427687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz" event={"ID":"a3b71b45-ca5c-41b6-b36c-95ea623d5ce5","Type":"ContainerDied","Data":"6511a4d296a176b358372a3fabbfe00efe6c96521edc8d366cc6d965f940ac4d"} Sep 10 19:22:53.427756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.427716 2576 scope.go:117] "RemoveContainer" containerID="07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe" Sep 10 19:22:53.453718 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.453281 2576 scope.go:117] "RemoveContainer" containerID="07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe" Sep 10 19:22:53.453811 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:22:53.453682 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe\": container with ID starting with 07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe not found: ID does not exist" containerID="07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe" Sep 10 19:22:53.453811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.453749 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe"} err="failed to get container status \"07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe\": rpc error: code = NotFound desc = could not find container \"07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe\": container with ID starting with 07dc6c0192cf64c0981ca7741a17753f338f02ad9f381f5ca8d593fe7fbf25fe not found: ID does not exist" Sep 10 19:22:53.475413 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.475367 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz"] Sep 10 19:22:53.480192 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.480161 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8db50f5a-5b4f8f8dfc-gk5dz"] Sep 10 19:22:53.483201 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.483181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:53.907927 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.907893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:53.995087 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:53.995052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:54.036462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.036019 2576 patch_prober.go:28] interesting pod/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 19:22:54.036462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.036078 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 19:22:54.094305 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.094250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:54.398249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.398216 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl"] Sep 10 19:22:54.398914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.398882 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" containerName="payload-processing" Sep 10 19:22:54.398914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.398912 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" containerName="payload-processing" Sep 10 19:22:54.399071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.398934 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb9e892-b079-46c3-a893-e89c996a59d6" containerName="revoke-keys" Sep 10 19:22:54.399071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.398943 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb9e892-b079-46c3-a893-e89c996a59d6" containerName="revoke-keys" Sep 10 19:22:54.399071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.399044 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb9e892-b079-46c3-a893-e89c996a59d6" containerName="revoke-keys" Sep 10 19:22:54.399071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.399060 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" containerName="payload-processing" Sep 10 19:22:54.404786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.404750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.407473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.407444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b59ba4a1-gw-openshift-default-dockercfg-72n5z\"" Sep 10 19:22:54.415950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.415922 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl"] Sep 10 19:22:54.439417 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.439383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:54.476855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.476821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-socket\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.476855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.476853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxdd\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-kube-api-access-mzxdd\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.477065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.476873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-credential-socket\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.477065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.476888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-podinfo\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.477065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.477004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-certs\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.477065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.477036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istiod-ca-cert\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.477065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.477057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-token\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.477304 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.477175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-envoy\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.477304 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.477226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-data\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578173 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-data\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-socket\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxdd\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-kube-api-access-mzxdd\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-credential-socket\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-podinfo\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-certs\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istiod-ca-cert\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-token\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-envoy\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.579086 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-data\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.579086 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-socket\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.579086 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578782 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-certs\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.579086 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.578853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-credential-socket\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.579232 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.579082 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istiod-ca-cert\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.580791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.580760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-envoy\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.581040 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.581019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-podinfo\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.586910 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.586889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-token\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.587184 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.587167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxdd\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-kube-api-access-mzxdd\") pod \"e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.603249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.603218 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b71b45-ca5c-41b6-b36c-95ea623d5ce5" path="/var/lib/kubelet/pods/a3b71b45-ca5c-41b6-b36c-95ea623d5ce5/volumes" Sep 10 19:22:54.603678 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.603659 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb9e892-b079-46c3-a893-e89c996a59d6" path="/var/lib/kubelet/pods/deb9e892-b079-46c3-a893-e89c996a59d6/volumes" Sep 10 19:22:54.717249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.717179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:54.857305 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.857271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:54.997812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:54.997788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl"] Sep 10 19:22:54.999404 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:22:54.999342 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8027a2db_826e_4cf5_a8d7_ffbeb16639ac.slice/crio-2d8c06eca3d61b22f8e6db9b04223f33270aa43871f1e2f44fb84b17f2309c96 WatchSource:0}: Error finding container 2d8c06eca3d61b22f8e6db9b04223f33270aa43871f1e2f44fb84b17f2309c96: Status 404 returned error can't find the container with id 2d8c06eca3d61b22f8e6db9b04223f33270aa43871f1e2f44fb84b17f2309c96 Sep 10 19:22:55.001659 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.001626 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:55.001748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.001691 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:55.001748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.001722 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:22:55.040916 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.040885 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:55.041310 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.041273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:55.042164 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.042146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:22:55.368673 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.368636 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:55.444251 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.444210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" event={"ID":"8027a2db-826e-4cf5-a8d7-ffbeb16639ac","Type":"ContainerStarted","Data":"384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6"} Sep 10 19:22:55.444442 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.444310 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" event={"ID":"8027a2db-826e-4cf5-a8d7-ffbeb16639ac","Type":"ContainerStarted","Data":"2d8c06eca3d61b22f8e6db9b04223f33270aa43871f1e2f44fb84b17f2309c96"} Sep 10 19:22:55.468028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.467980 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podStartSLOduration=1.467963639 podStartE2EDuration="1.467963639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:55.465662299 +0000 UTC m=+741.382318350" watchObservedRunningTime="2026-09-10 19:22:55.467963639 +0000 UTC m=+741.384619668" Sep 10 19:22:55.717739 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:55.717661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:56.722983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:56.722955 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:56.941968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:56.941927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:57.041920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:57.041892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:57.248510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:57.248479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:57.439813 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:57.439692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:57.456394 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:57.456358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:57.457474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:57.457455 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:22:57.541696 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:57.541659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:57.723069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:57.722990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:58.297058 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.297025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:58.502497 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.502463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:22:58.687036 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.687012 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:58.821512 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821484 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-credential-socket\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821528 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-envoy\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-socket\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821579 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhqvm\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-kube-api-access-fhqvm\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821606 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-token\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821680 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-certs\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821717 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-data\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821744 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d93171-bc91-43a9-890c-9eaadb36ee50-istiod-ca-cert\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821760 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-podinfo\") pod \"58d93171-bc91-43a9-890c-9eaadb36ee50\" (UID: \"58d93171-bc91-43a9-890c-9eaadb36ee50\") " Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821823 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:22:58.821920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821846 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:22:58.822478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.821978 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-data" (OuterVolumeSpecName: "istio-data") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:22:58.822478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.822151 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:22:58.822478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.822193 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.822478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.822214 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.822478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.822228 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.822478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.822224 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d93171-bc91-43a9-890c-9eaadb36ee50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:22:58.824182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.824152 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:22:58.824799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.824771 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:22:58.824895 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.824836 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-kube-api-access-fhqvm" (OuterVolumeSpecName: "kube-api-access-fhqvm") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "kube-api-access-fhqvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:22:58.824895 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.824860 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-token" (OuterVolumeSpecName: "istio-token") pod "58d93171-bc91-43a9-890c-9eaadb36ee50" (UID: "58d93171-bc91-43a9-890c-9eaadb36ee50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:22:58.923032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.922957 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d93171-bc91-43a9-890c-9eaadb36ee50-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.923032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.922984 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.923032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.922992 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.923032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.923000 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqvm\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-kube-api-access-fhqvm\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.923032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.923008 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d93171-bc91-43a9-890c-9eaadb36ee50-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:58.923032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:58.923018 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d93171-bc91-43a9-890c-9eaadb36ee50-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:22:59.470710 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.470675 2576 generic.go:356] "Generic (PLEG): container finished" podID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerID="ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a" exitCode=0 Sep 10 19:22:59.470904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.470730 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" Sep 10 19:22:59.470904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.470789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" event={"ID":"58d93171-bc91-43a9-890c-9eaadb36ee50","Type":"ContainerDied","Data":"ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a"} Sep 10 19:22:59.470904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.470849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq" event={"ID":"58d93171-bc91-43a9-890c-9eaadb36ee50","Type":"ContainerDied","Data":"a88bf12d05e50c24d14b9b9b010c5b36ca88966bfd88729a07b26a51b9ba9531"} Sep 10 19:22:59.470904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.470874 2576 scope.go:117] "RemoveContainer" containerID="ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a" Sep 10 19:22:59.481154 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.481139 2576 scope.go:117] "RemoveContainer" containerID="3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57" Sep 10 19:22:59.490197 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.490178 2576 scope.go:117] "RemoveContainer" containerID="ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a" Sep 10 19:22:59.490582 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:22:59.490560 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a\": container with ID starting with ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a not found: ID does not exist" containerID="ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a" Sep 10 19:22:59.490650 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.490591 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a"} err="failed to get container status \"ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a\": rpc error: code = NotFound desc = could not find container \"ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a\": container with ID starting with ecdc405c6f3a6e094e7e0ba24a7faaba222251277bfcb10e74200850eab1c74a not found: ID does not exist" Sep 10 19:22:59.490650 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.490610 2576 scope.go:117] "RemoveContainer" containerID="3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57" Sep 10 19:22:59.490864 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:22:59.490847 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57\": container with ID starting with 3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57 not found: ID does not exist" containerID="3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57" Sep 10 19:22:59.490909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.490871 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57"} err="failed to get container status \"3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57\": rpc error: code = NotFound desc = could not find container \"3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57\": container with ID starting with 3d96ad797b41523e8ce367c1a66f579e51991a08218b3f2f7a994518dcc1ea57 not found: ID does not exist" Sep 10 19:22:59.522931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.522909 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq"] Sep 10 19:22:59.537662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:22:59.537642 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8db50f5a-openshift-default-64c594b586-s76pq"] Sep 10 19:23:00.602634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:00.602600 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" path="/var/lib/kubelet/pods/58d93171-bc91-43a9-890c-9eaadb36ee50/volumes" Sep 10 19:23:03.670339 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:03.670305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:05.007831 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:05.007799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:08.042964 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:08.042928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:19.611574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:19.611543 2576 generic.go:356] "Generic (PLEG): container finished" podID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerID="384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6" exitCode=0 Sep 10 19:23:19.611991 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:19.611617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" event={"ID":"8027a2db-826e-4cf5-a8d7-ffbeb16639ac","Type":"ContainerDied","Data":"384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6"} Sep 10 19:23:19.612057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:19.612034 2576 scope.go:117] "RemoveContainer" containerID="384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6" Sep 10 19:23:19.613221 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:19.613192 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:19.613285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:19.613264 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:19.613327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:19.613298 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:20.618405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:20.618356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" event={"ID":"8027a2db-826e-4cf5-a8d7-ffbeb16639ac","Type":"ContainerStarted","Data":"bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d"} Sep 10 19:23:20.717995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:20.717967 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:23:20.718191 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:20.718168 2576 patch_prober.go:28] interesting pod/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 19:23:20.718242 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:20.718222 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 19:23:21.717828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.717793 2576 patch_prober.go:28] interesting pod/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 19:23:21.718207 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.717847 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 19:23:21.969055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.968974 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x"] Sep 10 19:23:21.969415 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.969402 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" Sep 10 19:23:21.969473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.969417 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" Sep 10 19:23:21.969473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.969435 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" Sep 10 19:23:21.969473 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.969441 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" Sep 10 19:23:21.969566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.969538 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" Sep 10 19:23:21.969566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.969552 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d93171-bc91-43a9-890c-9eaadb36ee50" containerName="istio-proxy" Sep 10 19:23:21.972184 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.972167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" Sep 10 19:23:21.975474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.975450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:23:21.976572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.976551 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:23:21.987364 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:21.987341 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x"] Sep 10 19:23:22.061999 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.061959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3922fe3d-7126-408c-a700-9346c3c8a0fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x\" (UID: \"3922fe3d-7126-408c-a700-9346c3c8a0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" Sep 10 19:23:22.163114 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.163075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3922fe3d-7126-408c-a700-9346c3c8a0fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x\" (UID: \"3922fe3d-7126-408c-a700-9346c3c8a0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" Sep 10 19:23:22.163784 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.163765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3922fe3d-7126-408c-a700-9346c3c8a0fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x\" (UID: \"3922fe3d-7126-408c-a700-9346c3c8a0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" Sep 10 19:23:22.282790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.282708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" Sep 10 19:23:22.584476 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.584451 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x"] Sep 10 19:23:22.586938 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:23:22.586909 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3922fe3d_7126_408c_a700_9346c3c8a0fb.slice/crio-fc646b8b7f97ee3494c9ef869f6e788f4f9bdce58a77c10bea980fd874601726 WatchSource:0}: Error finding container fc646b8b7f97ee3494c9ef869f6e788f4f9bdce58a77c10bea980fd874601726: Status 404 returned error can't find the container with id fc646b8b7f97ee3494c9ef869f6e788f4f9bdce58a77c10bea980fd874601726 Sep 10 19:23:22.630271 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.630237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" event={"ID":"3922fe3d-7126-408c-a700-9346c3c8a0fb","Type":"ContainerStarted","Data":"fc646b8b7f97ee3494c9ef869f6e788f4f9bdce58a77c10bea980fd874601726"} Sep 10 19:23:22.718057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.718021 2576 patch_prober.go:28] interesting pod/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 19:23:22.718563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:22.718083 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 19:23:23.663445 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:23.663403 2576 generic.go:356] "Generic (PLEG): container finished" podID="3922fe3d-7126-408c-a700-9346c3c8a0fb" containerID="98a43f89ff9a432070553153fc7fa19f63e1cf478d69e967223e8091d2bcf029" exitCode=0 Sep 10 19:23:23.663654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:23.663489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" event={"ID":"3922fe3d-7126-408c-a700-9346c3c8a0fb","Type":"ContainerDied","Data":"98a43f89ff9a432070553153fc7fa19f63e1cf478d69e967223e8091d2bcf029"} Sep 10 19:23:23.718625 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:23.718595 2576 patch_prober.go:28] interesting pod/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 19:23:23.718995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:23.718646 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 19:23:24.717993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:24.717961 2576 patch_prober.go:28] interesting pod/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 19:23:24.718152 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:24.718015 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 19:23:24.817538 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:24.817511 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" Sep 10 19:23:24.886429 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:24.886367 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3922fe3d-7126-408c-a700-9346c3c8a0fb-maas-api-service-ca\") pod \"3922fe3d-7126-408c-a700-9346c3c8a0fb\" (UID: \"3922fe3d-7126-408c-a700-9346c3c8a0fb\") " Sep 10 19:23:24.886655 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:24.886630 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3922fe3d-7126-408c-a700-9346c3c8a0fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3922fe3d-7126-408c-a700-9346c3c8a0fb" (UID: "3922fe3d-7126-408c-a700-9346c3c8a0fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:23:24.988112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:24.988012 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3922fe3d-7126-408c-a700-9346c3c8a0fb-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:25.676838 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:25.676808 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" Sep 10 19:23:25.676838 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:25.676832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x" event={"ID":"3922fe3d-7126-408c-a700-9346c3c8a0fb","Type":"ContainerDied","Data":"fc646b8b7f97ee3494c9ef869f6e788f4f9bdce58a77c10bea980fd874601726"} Sep 10 19:23:25.677057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:25.676884 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc646b8b7f97ee3494c9ef869f6e788f4f9bdce58a77c10bea980fd874601726" Sep 10 19:23:25.718068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:25.718035 2576 patch_prober.go:28] interesting pod/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 19:23:25.718222 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:25.718086 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 19:23:26.722196 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:26.722166 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:23:26.722600 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:26.722470 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:23:26.723203 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:26.723186 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:23:28.571080 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:28.571051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:28.702333 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:28.702291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:28.815078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:28.815047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:28.959096 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:28.959018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:28.972321 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:28.972292 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x"] Sep 10 19:23:28.978627 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:28.978599 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b59ba4a1-zqv4x"] Sep 10 19:23:29.112704 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.112675 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl"] Sep 10 19:23:29.118463 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.118440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:29.209708 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.209629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:29.547820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.547786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:29.699618 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.699577 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" containerID="cri-o://bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d" gracePeriod=30 Sep 10 19:23:29.867580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.867493 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz"] Sep 10 19:23:29.867973 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.867955 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3922fe3d-7126-408c-a700-9346c3c8a0fb" containerName="revoke-keys" Sep 10 19:23:29.868062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.867975 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3922fe3d-7126-408c-a700-9346c3c8a0fb" containerName="revoke-keys" Sep 10 19:23:29.868120 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.868104 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3922fe3d-7126-408c-a700-9346c3c8a0fb" containerName="revoke-keys" Sep 10 19:23:29.871162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.871142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.874825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.874804 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f21c165c-openshift-default-dockercfg-dsvsw\"" Sep 10 19:23:29.892760 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.892724 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz"] Sep 10 19:23:29.904488 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.904460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:29.934068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-envoy\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-credential-socket\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istiod-ca-cert\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-socket\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934404 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-data\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934404 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-token\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934404 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-certs\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934523 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-podinfo\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:29.934523 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:29.934443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9pd\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-kube-api-access-jq9pd\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035202 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-data\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035413 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-token\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035413 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-certs\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-podinfo\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9pd\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-kube-api-access-jq9pd\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-envoy\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035691 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-credential-socket\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035691 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-data\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035691 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istiod-ca-cert\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035691 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-certs\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035691 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-socket\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.035953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.035913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-credential-socket\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.036024 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.036005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-socket\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.036272 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.036251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istiod-ca-cert\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.037904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.037880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-envoy\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.038163 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.038145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-podinfo\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.044982 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.044959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-token\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.047272 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.047245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9pd\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-kube-api-access-jq9pd\") pod \"e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.072245 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.072206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:30.181701 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.181618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:30.468102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.468066 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz"] Sep 10 19:23:30.470539 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:23:30.470510 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa935a8_88b3_459e_be1e_e8dbedb5c85f.slice/crio-eda08e6d09269270690078dbadc9bb7d03bf0e2f344c129df1851b9bbcf1dd1e WatchSource:0}: Error finding container eda08e6d09269270690078dbadc9bb7d03bf0e2f344c129df1851b9bbcf1dd1e: Status 404 returned error can't find the container with id eda08e6d09269270690078dbadc9bb7d03bf0e2f344c129df1851b9bbcf1dd1e Sep 10 19:23:30.472847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.472812 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:30.472935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.472877 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:30.472935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.472909 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:30.603603 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.603571 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3922fe3d-7126-408c-a700-9346c3c8a0fb" path="/var/lib/kubelet/pods/3922fe3d-7126-408c-a700-9346c3c8a0fb/volumes" Sep 10 19:23:30.707054 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.707024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" event={"ID":"2fa935a8-88b3-459e-be1e-e8dbedb5c85f","Type":"ContainerStarted","Data":"aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b"} Sep 10 19:23:30.707494 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:30.707084 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" event={"ID":"2fa935a8-88b3-459e-be1e-e8dbedb5c85f","Type":"ContainerStarted","Data":"eda08e6d09269270690078dbadc9bb7d03bf0e2f344c129df1851b9bbcf1dd1e"} Sep 10 19:23:31.181989 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:31.181957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:31.187555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:31.187529 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:31.214095 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:31.214049 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podStartSLOduration=2.214035417 podStartE2EDuration="2.214035417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:23:30.746565519 +0000 UTC m=+776.663221553" watchObservedRunningTime="2026-09-10 19:23:31.214035417 +0000 UTC m=+777.130691446" Sep 10 19:23:31.713120 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:31.713093 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:31.714103 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:31.714083 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:32.114577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:32.114543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:32.244671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:32.244634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:33.461170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:33.461110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:33.563985 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:33.563919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:34.018584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:34.018553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:34.341652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:34.341607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:34.460839 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:34.460805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:34.968510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:34.968441 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:23:35.084954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.084925 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzxdd\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-kube-api-access-mzxdd\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.084972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-certs\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085002 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istiod-ca-cert\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-token\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085061 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-socket\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-credential-socket\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-data\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085171 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-envoy\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085204 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-podinfo\") pod \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\" (UID: \"8027a2db-826e-4cf5-a8d7-ffbeb16639ac\") " Sep 10 19:23:35.085502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085276 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:23:35.085669 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085514 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:23:35.085669 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085578 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.085669 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085595 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:23:35.085669 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085536 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:23:35.085816 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.085727 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:23:35.087563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.087527 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:23:35.087563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.087537 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-kube-api-access-mzxdd" (OuterVolumeSpecName: "kube-api-access-mzxdd") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "kube-api-access-mzxdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:23:35.087694 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.087568 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:23:35.087694 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.087623 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8027a2db-826e-4cf5-a8d7-ffbeb16639ac" (UID: "8027a2db-826e-4cf5-a8d7-ffbeb16639ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:23:35.186288 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186252 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.186288 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186281 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.186288 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186290 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.186288 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186298 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.186572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186307 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.186572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186315 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxdd\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-kube-api-access-mzxdd\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.186572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186323 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.186572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.186331 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8027a2db-826e-4cf5-a8d7-ffbeb16639ac-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:23:35.737695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.737601 2576 generic.go:356] "Generic (PLEG): container finished" podID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerID="bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d" exitCode=0 Sep 10 19:23:35.737695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.737654 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" Sep 10 19:23:35.737695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.737679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" event={"ID":"8027a2db-826e-4cf5-a8d7-ffbeb16639ac","Type":"ContainerDied","Data":"bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d"} Sep 10 19:23:35.737967 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.737731 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl" event={"ID":"8027a2db-826e-4cf5-a8d7-ffbeb16639ac","Type":"ContainerDied","Data":"2d8c06eca3d61b22f8e6db9b04223f33270aa43871f1e2f44fb84b17f2309c96"} Sep 10 19:23:35.737967 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.737753 2576 scope.go:117] "RemoveContainer" containerID="bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d" Sep 10 19:23:35.748140 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.748124 2576 scope.go:117] "RemoveContainer" containerID="384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6" Sep 10 19:23:35.756204 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.756187 2576 scope.go:117] "RemoveContainer" containerID="bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d" Sep 10 19:23:35.756492 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:23:35.756474 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d\": container with ID starting with bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d not found: ID does not exist" containerID="bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d" Sep 10 19:23:35.756555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.756500 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d"} err="failed to get container status \"bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d\": rpc error: code = NotFound desc = could not find container \"bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d\": container with ID starting with bb7ccc25cb1c181820e72bdf7cae86c8647ad741e75a01a84c848834ea6c4d4d not found: ID does not exist" Sep 10 19:23:35.756555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.756517 2576 scope.go:117] "RemoveContainer" containerID="384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6" Sep 10 19:23:35.756784 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:23:35.756766 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6\": container with ID starting with 384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6 not found: ID does not exist" containerID="384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6" Sep 10 19:23:35.756835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.756789 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6"} err="failed to get container status \"384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6\": rpc error: code = NotFound desc = could not find container \"384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6\": container with ID starting with 384d9063b50625f3dcf39ab04aefc9880972f8e4741e617808e2e68d89b113c6 not found: ID does not exist" Sep 10 19:23:35.762285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.762266 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl"] Sep 10 19:23:35.765915 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:35.765888 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b59ba4a1-gw-openshift-default-788f6dbf5b-p6dgl"] Sep 10 19:23:36.602820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:36.602787 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" path="/var/lib/kubelet/pods/8027a2db-826e-4cf5-a8d7-ffbeb16639ac/volumes" Sep 10 19:23:39.525000 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:39.524967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:23:54.847434 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:54.847401 2576 generic.go:356] "Generic (PLEG): container finished" podID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerID="aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b" exitCode=0 Sep 10 19:23:54.847855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:54.847474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" event={"ID":"2fa935a8-88b3-459e-be1e-e8dbedb5c85f","Type":"ContainerDied","Data":"aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b"} Sep 10 19:23:54.847959 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:54.847944 2576 scope.go:117] "RemoveContainer" containerID="aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b" Sep 10 19:23:54.849106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:54.849080 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:54.849180 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:54.849147 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:54.849234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:54.849180 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:23:55.859445 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:55.859408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" event={"ID":"2fa935a8-88b3-459e-be1e-e8dbedb5c85f","Type":"ContainerStarted","Data":"020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7"} Sep 10 19:23:56.182435 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:56.182323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:23:56.182636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:56.182615 2576 patch_prober.go:28] interesting pod/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 19:23:56.182684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:56.182659 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 19:23:57.182649 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.182614 2576 patch_prober.go:28] interesting pod/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 19:23:57.183040 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.182682 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 19:23:57.838162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.838125 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7"] Sep 10 19:23:57.839047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.839021 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" Sep 10 19:23:57.839199 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.839184 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" Sep 10 19:23:57.839327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.839314 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" Sep 10 19:23:57.839464 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.839452 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" Sep 10 19:23:57.839692 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.839678 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" Sep 10 19:23:57.839799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.839787 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8027a2db-826e-4cf5-a8d7-ffbeb16639ac" containerName="istio-proxy" Sep 10 19:23:57.843465 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.843446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" Sep 10 19:23:57.847753 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.847732 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:23:57.848016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.847991 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:23:57.849555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:57.849526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7"] Sep 10 19:23:58.014848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.014812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9705a5a2-434b-45ef-a609-0be43930693a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7\" (UID: \"9705a5a2-434b-45ef-a609-0be43930693a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" Sep 10 19:23:58.116267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.116179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9705a5a2-434b-45ef-a609-0be43930693a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7\" (UID: \"9705a5a2-434b-45ef-a609-0be43930693a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" Sep 10 19:23:58.116872 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.116850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9705a5a2-434b-45ef-a609-0be43930693a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7\" (UID: \"9705a5a2-434b-45ef-a609-0be43930693a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" Sep 10 19:23:58.165315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.165278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" Sep 10 19:23:58.182656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.182625 2576 patch_prober.go:28] interesting pod/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 19:23:58.182778 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.182691 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 19:23:58.455833 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.444227 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7"] Sep 10 19:23:58.881652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.881609 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" event={"ID":"9705a5a2-434b-45ef-a609-0be43930693a","Type":"ContainerStarted","Data":"c1ce208791d2d56b48fd0e3fd4224c50f3ac7e542ff54ae5971f9b8649194142"} Sep 10 19:23:58.881845 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:58.881671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" event={"ID":"9705a5a2-434b-45ef-a609-0be43930693a","Type":"ContainerStarted","Data":"24734059d5c445e6153ac3006584b49b25c21dfe5c4636b4278512fee6584697"} Sep 10 19:23:59.182754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:59.182717 2576 patch_prober.go:28] interesting pod/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 19:23:59.182957 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:59.182792 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 19:23:59.893974 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:59.893941 2576 generic.go:356] "Generic (PLEG): container finished" podID="9705a5a2-434b-45ef-a609-0be43930693a" containerID="c1ce208791d2d56b48fd0e3fd4224c50f3ac7e542ff54ae5971f9b8649194142" exitCode=0 Sep 10 19:23:59.894158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:23:59.893980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" event={"ID":"9705a5a2-434b-45ef-a609-0be43930693a","Type":"ContainerDied","Data":"c1ce208791d2d56b48fd0e3fd4224c50f3ac7e542ff54ae5971f9b8649194142"} Sep 10 19:24:00.182639 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:00.182554 2576 patch_prober.go:28] interesting pod/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 19:24:00.182639 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:00.182614 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 19:24:01.080884 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.080858 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" Sep 10 19:24:01.182330 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.182233 2576 patch_prober.go:28] interesting pod/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 19:24:01.182330 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.182295 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 19:24:01.259090 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.259057 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9705a5a2-434b-45ef-a609-0be43930693a-maas-api-service-ca\") pod \"9705a5a2-434b-45ef-a609-0be43930693a\" (UID: \"9705a5a2-434b-45ef-a609-0be43930693a\") " Sep 10 19:24:01.259464 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.259437 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9705a5a2-434b-45ef-a609-0be43930693a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9705a5a2-434b-45ef-a609-0be43930693a" (UID: "9705a5a2-434b-45ef-a609-0be43930693a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:01.360219 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.360183 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9705a5a2-434b-45ef-a609-0be43930693a-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:01.713458 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.713424 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:24:01.907480 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.907442 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" Sep 10 19:24:01.907647 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.907477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7" event={"ID":"9705a5a2-434b-45ef-a609-0be43930693a","Type":"ContainerDied","Data":"24734059d5c445e6153ac3006584b49b25c21dfe5c4636b4278512fee6584697"} Sep 10 19:24:01.907647 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:01.907523 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24734059d5c445e6153ac3006584b49b25c21dfe5c4636b4278512fee6584697" Sep 10 19:24:02.186303 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:02.186260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:24:02.187363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:02.187345 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:24:04.055325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:04.055292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:04.155195 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:04.155162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:04.248310 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:04.248278 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7"] Sep 10 19:24:04.257117 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:04.257085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:04.257751 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:04.257733 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f21c165c-zl4w7"] Sep 10 19:24:04.334191 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:04.334102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:04.602665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:04.602572 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9705a5a2-434b-45ef-a609-0be43930693a" path="/var/lib/kubelet/pods/9705a5a2-434b-45ef-a609-0be43930693a/volumes" Sep 10 19:24:05.380295 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:05.380266 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz"] Sep 10 19:24:05.380757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:05.380473 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" containerID="cri-o://020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7" gracePeriod=30 Sep 10 19:24:05.403587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:05.403553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:05.833482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:05.833450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:06.124860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.124787 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w"] Sep 10 19:24:06.125180 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.125168 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9705a5a2-434b-45ef-a609-0be43930693a" containerName="revoke-keys" Sep 10 19:24:06.125225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.125182 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9705a5a2-434b-45ef-a609-0be43930693a" containerName="revoke-keys" Sep 10 19:24:06.125257 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.125249 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9705a5a2-434b-45ef-a609-0be43930693a" containerName="revoke-keys" Sep 10 19:24:06.128643 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.128627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.131411 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.131358 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a888f7aa-openshift-default-dockercfg-2dx29\"" Sep 10 19:24:06.141727 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.141705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w"] Sep 10 19:24:06.154977 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.154953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:06.207405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.207299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-credential-socket\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.207738 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.207706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a21f16-293f-4a69-b781-2025c62bf661-istio-podinfo\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.207889 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.207753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-socket\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.207889 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.207812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-certs\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.207889 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.207863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-envoy\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.208101 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.207957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktsh\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-kube-api-access-fktsh\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.208101 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.208023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-data\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.208101 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.208050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a21f16-293f-4a69-b781-2025c62bf661-istiod-ca-cert\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.208101 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.208089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-istio-token\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.308709 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.308674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktsh\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-kube-api-access-fktsh\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.308912 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.308726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-data\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.308912 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.308851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a21f16-293f-4a69-b781-2025c62bf661-istiod-ca-cert\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.308912 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.308897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-istio-token\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.308938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-credential-socket\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a21f16-293f-4a69-b781-2025c62bf661-istio-podinfo\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-data\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-socket\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-certs\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-envoy\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-credential-socket\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309529 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-socket\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309529 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-certs\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.309616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.309586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a21f16-293f-4a69-b781-2025c62bf661-istiod-ca-cert\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.311461 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.311441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-envoy\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.311559 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.311549 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a21f16-293f-4a69-b781-2025c62bf661-istio-podinfo\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.317899 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.317875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-istio-token\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.318009 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.317911 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktsh\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-kube-api-access-fktsh\") pod \"e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.435448 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.435349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:06.439560 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.439545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:06.727968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.727945 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w"] Sep 10 19:24:06.729887 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:24:06.729859 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a21f16_293f_4a69_b781_2025c62bf661.slice/crio-764479e306386e925d8048723cf3850b3e557b123cc9d68922e33486dea31553 WatchSource:0}: Error finding container 764479e306386e925d8048723cf3850b3e557b123cc9d68922e33486dea31553: Status 404 returned error can't find the container with id 764479e306386e925d8048723cf3850b3e557b123cc9d68922e33486dea31553 Sep 10 19:24:06.732725 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.732683 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:24:06.732827 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.732780 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:24:06.732873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.732823 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:24:06.939630 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.939597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" event={"ID":"a4a21f16-293f-4a69-b781-2025c62bf661","Type":"ContainerStarted","Data":"11bf4c423ac414b4c4b5bcdd6614d835e01a8bcc09ab702d633fd26f059f3f08"} Sep 10 19:24:06.939833 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.939656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" event={"ID":"a4a21f16-293f-4a69-b781-2025c62bf661","Type":"ContainerStarted","Data":"764479e306386e925d8048723cf3850b3e557b123cc9d68922e33486dea31553"} Sep 10 19:24:06.961555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:06.961502 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" podStartSLOduration=0.961483499 podStartE2EDuration="961.483499ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:06.959934972 +0000 UTC m=+812.876591001" watchObservedRunningTime="2026-09-10 19:24:06.961483499 +0000 UTC m=+812.878139529" Sep 10 19:24:07.440141 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:07.440110 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:07.445261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:07.445235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:07.945876 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:07.945840 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:07.946750 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:07.946730 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:08.339337 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:08.339304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:08.433914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:08.433882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:09.578933 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:09.578900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:09.673330 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:09.673300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:10.394838 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.394799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:10.535215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.535175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:10.647410 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.647293 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:24:10.715399 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.714224 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb"] Sep 10 19:24:10.715399 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.714836 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" Sep 10 19:24:10.715399 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.714855 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" Sep 10 19:24:10.715399 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.714998 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" Sep 10 19:24:10.715399 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.715011 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" Sep 10 19:24:10.718496 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.718475 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" Sep 10 19:24:10.721450 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.721429 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:24:10.722526 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.722506 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:24:10.729119 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.729045 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb"] Sep 10 19:24:10.752931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.752897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-certs\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753092 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.752947 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-token\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753092 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.752966 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-credential-socket\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753092 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.752981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-data\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753092 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753013 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-envoy\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753092 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istiod-ca-cert\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753410 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-socket\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753410 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753152 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq9pd\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-kube-api-access-jq9pd\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753410 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753203 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-podinfo\") pod \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\" (UID: \"2fa935a8-88b3-459e-be1e-e8dbedb5c85f\") " Sep 10 19:24:10.753410 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753245 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:10.753410 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753271 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:10.753665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753474 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:10.753665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7398ead3-174f-46e3-b211-2a98b6cf0160-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb\" (UID: \"7398ead3-174f-46e3-b211-2a98b6cf0160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" Sep 10 19:24:10.753665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753599 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.753665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753615 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.753665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753618 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:10.753665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753629 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.753969 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.753875 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:10.756437 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.756409 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:24:10.756567 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.756446 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:24:10.756634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.756563 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-kube-api-access-jq9pd" (OuterVolumeSpecName: "kube-api-access-jq9pd") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "kube-api-access-jq9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:24:10.756797 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.756765 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2fa935a8-88b3-459e-be1e-e8dbedb5c85f" (UID: "2fa935a8-88b3-459e-be1e-e8dbedb5c85f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:10.854791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.854750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7398ead3-174f-46e3-b211-2a98b6cf0160-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb\" (UID: \"7398ead3-174f-46e3-b211-2a98b6cf0160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" Sep 10 19:24:10.854966 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.854919 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.854966 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.854936 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.854966 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.854949 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.854966 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.854963 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.855144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.854976 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9pd\" (UniqueName: \"kubernetes.io/projected/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-kube-api-access-jq9pd\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.855144 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.854990 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa935a8-88b3-459e-be1e-e8dbedb5c85f-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:10.855446 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.855419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7398ead3-174f-46e3-b211-2a98b6cf0160-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb\" (UID: \"7398ead3-174f-46e3-b211-2a98b6cf0160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" Sep 10 19:24:10.964821 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.964725 2576 generic.go:356] "Generic (PLEG): container finished" podID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerID="020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7" exitCode=0 Sep 10 19:24:10.964821 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.964784 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" Sep 10 19:24:10.964821 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.964794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" event={"ID":"2fa935a8-88b3-459e-be1e-e8dbedb5c85f","Type":"ContainerDied","Data":"020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7"} Sep 10 19:24:10.965036 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.964847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz" event={"ID":"2fa935a8-88b3-459e-be1e-e8dbedb5c85f","Type":"ContainerDied","Data":"eda08e6d09269270690078dbadc9bb7d03bf0e2f344c129df1851b9bbcf1dd1e"} Sep 10 19:24:10.965036 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.964876 2576 scope.go:117] "RemoveContainer" containerID="020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7" Sep 10 19:24:10.977285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.977265 2576 scope.go:117] "RemoveContainer" containerID="aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b" Sep 10 19:24:10.986308 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.986289 2576 scope.go:117] "RemoveContainer" containerID="020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7" Sep 10 19:24:10.986603 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:24:10.986584 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7\": container with ID starting with 020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7 not found: ID does not exist" containerID="020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7" Sep 10 19:24:10.986684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.986615 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7"} err="failed to get container status \"020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7\": rpc error: code = NotFound desc = could not find container \"020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7\": container with ID starting with 020c23830ec3d61c8421d119e8c9d12dcce012c891b7468b5610f0d6dc1b55f7 not found: ID does not exist" Sep 10 19:24:10.986684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.986639 2576 scope.go:117] "RemoveContainer" containerID="aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b" Sep 10 19:24:10.986902 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:24:10.986882 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b\": container with ID starting with aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b not found: ID does not exist" containerID="aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b" Sep 10 19:24:10.986941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.986910 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b"} err="failed to get container status \"aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b\": rpc error: code = NotFound desc = could not find container \"aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b\": container with ID starting with aa486ae7ea223022684f79f940009dbc5259e2ff9674df440e9096313df4428b not found: ID does not exist" Sep 10 19:24:10.993983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.993759 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz"] Sep 10 19:24:10.997855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:10.997836 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f21c165c-openshift-default-bf447dc66-dhqlz"] Sep 10 19:24:11.031016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:11.030986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" Sep 10 19:24:11.129935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:11.129899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:11.323696 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:11.323670 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb"] Sep 10 19:24:11.325084 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:24:11.325063 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7398ead3_174f_46e3_b211_2a98b6cf0160.slice/crio-a902824f5b5e0d1b36d792b8a4e7011b94465584c6d4a4d8d88de6e8d5e0c5c7 WatchSource:0}: Error finding container a902824f5b5e0d1b36d792b8a4e7011b94465584c6d4a4d8d88de6e8d5e0c5c7: Status 404 returned error can't find the container with id a902824f5b5e0d1b36d792b8a4e7011b94465584c6d4a4d8d88de6e8d5e0c5c7 Sep 10 19:24:11.972060 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:11.971969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" event={"ID":"7398ead3-174f-46e3-b211-2a98b6cf0160","Type":"ContainerStarted","Data":"106d34c61662a726742aaa2868d84829f513a2dcc85ad5fb9bcb97bb984d783f"} Sep 10 19:24:11.972060 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:11.972030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" event={"ID":"7398ead3-174f-46e3-b211-2a98b6cf0160","Type":"ContainerStarted","Data":"a902824f5b5e0d1b36d792b8a4e7011b94465584c6d4a4d8d88de6e8d5e0c5c7"} Sep 10 19:24:11.993040 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:11.992994 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" podStartSLOduration=1.9929773 podStartE2EDuration="1.9929773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:11.991685979 +0000 UTC m=+817.908342009" watchObservedRunningTime="2026-09-10 19:24:11.9929773 +0000 UTC m=+817.909633328" Sep 10 19:24:12.074748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:12.074711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:12.603107 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:12.603068 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" path="/var/lib/kubelet/pods/2fa935a8-88b3-459e-be1e-e8dbedb5c85f/volumes" Sep 10 19:24:12.980190 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:12.980101 2576 generic.go:356] "Generic (PLEG): container finished" podID="7398ead3-174f-46e3-b211-2a98b6cf0160" containerID="106d34c61662a726742aaa2868d84829f513a2dcc85ad5fb9bcb97bb984d783f" exitCode=0 Sep 10 19:24:12.980586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:12.980183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" event={"ID":"7398ead3-174f-46e3-b211-2a98b6cf0160","Type":"ContainerDied","Data":"106d34c61662a726742aaa2868d84829f513a2dcc85ad5fb9bcb97bb984d783f"} Sep 10 19:24:14.126900 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:14.126876 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" Sep 10 19:24:14.184836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:14.184806 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7398ead3-174f-46e3-b211-2a98b6cf0160-maas-api-service-ca\") pod \"7398ead3-174f-46e3-b211-2a98b6cf0160\" (UID: \"7398ead3-174f-46e3-b211-2a98b6cf0160\") " Sep 10 19:24:14.185160 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:14.185138 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7398ead3-174f-46e3-b211-2a98b6cf0160-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7398ead3-174f-46e3-b211-2a98b6cf0160" (UID: "7398ead3-174f-46e3-b211-2a98b6cf0160"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:14.286434 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:14.286409 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7398ead3-174f-46e3-b211-2a98b6cf0160-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:14.992965 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:14.992887 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" Sep 10 19:24:14.993099 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:14.992891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb" event={"ID":"7398ead3-174f-46e3-b211-2a98b6cf0160","Type":"ContainerDied","Data":"a902824f5b5e0d1b36d792b8a4e7011b94465584c6d4a4d8d88de6e8d5e0c5c7"} Sep 10 19:24:14.993099 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:14.993010 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a902824f5b5e0d1b36d792b8a4e7011b94465584c6d4a4d8d88de6e8d5e0c5c7" Sep 10 19:24:20.945429 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:20.945313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:21.930982 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:21.930948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:22.022999 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.022971 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp"] Sep 10 19:24:22.023409 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.023203 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" podUID="7db16422-fc7a-46a5-bed9-7a2b3aa34868" containerName="payload-pre-processing" containerID="cri-o://ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29" gracePeriod=30 Sep 10 19:24:22.077432 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.075913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:22.165701 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.165671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:22.290990 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.290919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:22.362017 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.361481 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb"] Sep 10 19:24:22.364801 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.364767 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c7a5e-bmvrb"] Sep 10 19:24:22.365310 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.365293 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:24:22.413496 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.413466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:22.471044 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.470946 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db16422-fc7a-46a5-bed9-7a2b3aa34868-plugins-config-volume\") pod \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " Sep 10 19:24:22.471044 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.470997 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x72jh\" (UniqueName: \"kubernetes.io/projected/7db16422-fc7a-46a5-bed9-7a2b3aa34868-kube-api-access-x72jh\") pod \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\" (UID: \"7db16422-fc7a-46a5-bed9-7a2b3aa34868\") " Sep 10 19:24:22.471405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.471381 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db16422-fc7a-46a5-bed9-7a2b3aa34868-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7db16422-fc7a-46a5-bed9-7a2b3aa34868" (UID: "7db16422-fc7a-46a5-bed9-7a2b3aa34868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:22.473357 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.473334 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db16422-fc7a-46a5-bed9-7a2b3aa34868-kube-api-access-x72jh" (OuterVolumeSpecName: "kube-api-access-x72jh") pod "7db16422-fc7a-46a5-bed9-7a2b3aa34868" (UID: "7db16422-fc7a-46a5-bed9-7a2b3aa34868"). InnerVolumeSpecName "kube-api-access-x72jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:24:22.572593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.572558 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db16422-fc7a-46a5-bed9-7a2b3aa34868-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:22.572593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.572587 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72jh\" (UniqueName: \"kubernetes.io/projected/7db16422-fc7a-46a5-bed9-7a2b3aa34868-kube-api-access-x72jh\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:22.602681 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.602651 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7398ead3-174f-46e3-b211-2a98b6cf0160" path="/var/lib/kubelet/pods/7398ead3-174f-46e3-b211-2a98b6cf0160/volumes" Sep 10 19:24:22.707222 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:22.707187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:23.043145 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.043104 2576 generic.go:356] "Generic (PLEG): container finished" podID="7db16422-fc7a-46a5-bed9-7a2b3aa34868" containerID="ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29" exitCode=0 Sep 10 19:24:23.043624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.043161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" event={"ID":"7db16422-fc7a-46a5-bed9-7a2b3aa34868","Type":"ContainerDied","Data":"ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29"} Sep 10 19:24:23.043624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.043214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" event={"ID":"7db16422-fc7a-46a5-bed9-7a2b3aa34868","Type":"ContainerDied","Data":"9cd52894219d127a6672e843c20f33dfcc5ee8a80fdec0afc7b1d8bee251b703"} Sep 10 19:24:23.043624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.043246 2576 scope.go:117] "RemoveContainer" containerID="ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29" Sep 10 19:24:23.043624 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.043436 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp" Sep 10 19:24:23.062196 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.062173 2576 scope.go:117] "RemoveContainer" containerID="ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29" Sep 10 19:24:23.062644 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:24:23.062618 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29\": container with ID starting with ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29 not found: ID does not exist" containerID="ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29" Sep 10 19:24:23.062722 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.062656 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29"} err="failed to get container status \"ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29\": rpc error: code = NotFound desc = could not find container \"ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29\": container with ID starting with ad7423a9ebe143573cc3b98c5ccafabde6f7cc5db444136c4aa480acc5b89b29 not found: ID does not exist" Sep 10 19:24:23.088143 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.088117 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp"] Sep 10 19:24:23.106421 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:23.106395 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c7a5e-7bd54ff7d-7plnp"] Sep 10 19:24:24.602391 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:24.602334 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db16422-fc7a-46a5-bed9-7a2b3aa34868" path="/var/lib/kubelet/pods/7db16422-fc7a-46a5-bed9-7a2b3aa34868/volumes" Sep 10 19:24:27.783272 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:27.783184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:28.080768 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:28.080736 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4a21f16-293f-4a69-b781-2025c62bf661" containerID="11bf4c423ac414b4c4b5bcdd6614d835e01a8bcc09ab702d633fd26f059f3f08" exitCode=0 Sep 10 19:24:28.080946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:28.080782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" event={"ID":"a4a21f16-293f-4a69-b781-2025c62bf661","Type":"ContainerDied","Data":"11bf4c423ac414b4c4b5bcdd6614d835e01a8bcc09ab702d633fd26f059f3f08"} Sep 10 19:24:28.081159 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:28.081146 2576 scope.go:117] "RemoveContainer" containerID="11bf4c423ac414b4c4b5bcdd6614d835e01a8bcc09ab702d633fd26f059f3f08" Sep 10 19:24:28.082205 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:28.082179 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:24:28.082253 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:28.082245 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:24:28.082291 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:28.082277 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:24:28.798300 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:28.798260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:29.089021 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:29.088938 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" event={"ID":"a4a21f16-293f-4a69-b781-2025c62bf661","Type":"ContainerStarted","Data":"f5cfb0f1e59e6833c0669f125261dd6b289a2020335a9c88c5b1a576f86bf285"} Sep 10 19:24:29.440763 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:29.440659 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:29.440958 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:29.440801 2576 patch_prober.go:28] interesting pod/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 19:24:29.440958 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:29.440847 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 19:24:30.440732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:30.440693 2576 patch_prober.go:28] interesting pod/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 19:24:30.441332 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:30.441298 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 19:24:30.896919 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:30.896888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:31.441043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:31.441008 2576 patch_prober.go:28] interesting pod/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 19:24:31.441660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:31.441628 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 19:24:32.440800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:32.440751 2576 patch_prober.go:28] interesting pod/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 19:24:32.441005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:32.440833 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 19:24:33.183621 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.183589 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8"] Sep 10 19:24:33.184003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.183990 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" Sep 10 19:24:33.184003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.184002 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa935a8-88b3-459e-be1e-e8dbedb5c85f" containerName="istio-proxy" Sep 10 19:24:33.184084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.184012 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db16422-fc7a-46a5-bed9-7a2b3aa34868" containerName="payload-pre-processing" Sep 10 19:24:33.184084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.184017 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db16422-fc7a-46a5-bed9-7a2b3aa34868" containerName="payload-pre-processing" Sep 10 19:24:33.184084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.184028 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7398ead3-174f-46e3-b211-2a98b6cf0160" containerName="revoke-keys" Sep 10 19:24:33.184084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.184034 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7398ead3-174f-46e3-b211-2a98b6cf0160" containerName="revoke-keys" Sep 10 19:24:33.184205 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.184101 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db16422-fc7a-46a5-bed9-7a2b3aa34868" containerName="payload-pre-processing" Sep 10 19:24:33.184205 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.184110 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7398ead3-174f-46e3-b211-2a98b6cf0160" containerName="revoke-keys" Sep 10 19:24:33.187044 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.187027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" Sep 10 19:24:33.190076 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.190053 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:24:33.190194 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.190178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:24:33.195189 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.195167 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8"] Sep 10 19:24:33.278255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.278219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e12cac-fb22-437a-8353-f4ce0cfc0f0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8\" (UID: \"00e12cac-fb22-437a-8353-f4ce0cfc0f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" Sep 10 19:24:33.379011 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.378958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e12cac-fb22-437a-8353-f4ce0cfc0f0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8\" (UID: \"00e12cac-fb22-437a-8353-f4ce0cfc0f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" Sep 10 19:24:33.379748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.379718 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e12cac-fb22-437a-8353-f4ce0cfc0f0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8\" (UID: \"00e12cac-fb22-437a-8353-f4ce0cfc0f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" Sep 10 19:24:33.440619 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.440516 2576 patch_prober.go:28] interesting pod/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 10 19:24:33.440619 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.440589 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 10 19:24:33.498511 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.498471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" Sep 10 19:24:33.792396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:33.792357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8"] Sep 10 19:24:33.794265 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:24:33.794238 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e12cac_fb22_437a_8353_f4ce0cfc0f0b.slice/crio-cad56a8ee35013cef45c14e73104b419172de5afe20fccdd98bc8da50e4f60c6 WatchSource:0}: Error finding container cad56a8ee35013cef45c14e73104b419172de5afe20fccdd98bc8da50e4f60c6: Status 404 returned error can't find the container with id cad56a8ee35013cef45c14e73104b419172de5afe20fccdd98bc8da50e4f60c6 Sep 10 19:24:34.068323 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:34.068292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:34.119159 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:34.119126 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" event={"ID":"00e12cac-fb22-437a-8353-f4ce0cfc0f0b","Type":"ContainerStarted","Data":"cad56a8ee35013cef45c14e73104b419172de5afe20fccdd98bc8da50e4f60c6"} Sep 10 19:24:34.444681 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:34.444598 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:34.445058 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:34.444998 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:34.445749 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:34.445728 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:35.126236 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:35.126197 2576 generic.go:356] "Generic (PLEG): container finished" podID="00e12cac-fb22-437a-8353-f4ce0cfc0f0b" containerID="81e98c9e75b3ca62c5e17ae34349599ff792f5937f842edc6e38218cd5739913" exitCode=0 Sep 10 19:24:35.126475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:35.126288 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" event={"ID":"00e12cac-fb22-437a-8353-f4ce0cfc0f0b","Type":"ContainerDied","Data":"81e98c9e75b3ca62c5e17ae34349599ff792f5937f842edc6e38218cd5739913"} Sep 10 19:24:36.264914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:36.264892 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" Sep 10 19:24:36.406762 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:36.406674 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e12cac-fb22-437a-8353-f4ce0cfc0f0b-maas-api-service-ca\") pod \"00e12cac-fb22-437a-8353-f4ce0cfc0f0b\" (UID: \"00e12cac-fb22-437a-8353-f4ce0cfc0f0b\") " Sep 10 19:24:36.407004 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:36.406983 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e12cac-fb22-437a-8353-f4ce0cfc0f0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00e12cac-fb22-437a-8353-f4ce0cfc0f0b" (UID: "00e12cac-fb22-437a-8353-f4ce0cfc0f0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:36.507335 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:36.507300 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e12cac-fb22-437a-8353-f4ce0cfc0f0b-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:37.139136 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:37.139105 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" Sep 10 19:24:37.139136 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:37.139130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8" event={"ID":"00e12cac-fb22-437a-8353-f4ce0cfc0f0b","Type":"ContainerDied","Data":"cad56a8ee35013cef45c14e73104b419172de5afe20fccdd98bc8da50e4f60c6"} Sep 10 19:24:37.139414 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:37.139171 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad56a8ee35013cef45c14e73104b419172de5afe20fccdd98bc8da50e4f60c6" Sep 10 19:24:40.506620 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:40.506588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:40.605628 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:40.605588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:40.703724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:40.703692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:40.810087 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:40.810055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:40.836134 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:40.836104 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8"] Sep 10 19:24:40.839435 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:40.839414 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a888f7aa-xtxm8"] Sep 10 19:24:40.909131 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:40.909098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:41.019780 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:41.019753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w"] Sep 10 19:24:41.020005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:41.019982 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" containerID="cri-o://f5cfb0f1e59e6833c0669f125261dd6b289a2020335a9c88c5b1a576f86bf285" gracePeriod=30 Sep 10 19:24:41.032111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:41.032085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:41.717644 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:41.717611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:41.821803 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:41.821771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:42.602258 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:42.602221 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e12cac-fb22-437a-8353-f4ce0cfc0f0b" path="/var/lib/kubelet/pods/00e12cac-fb22-437a-8353-f4ce0cfc0f0b/volumes" Sep 10 19:24:43.305803 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:43.305768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:46.204907 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.204878 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4a21f16-293f-4a69-b781-2025c62bf661" containerID="f5cfb0f1e59e6833c0669f125261dd6b289a2020335a9c88c5b1a576f86bf285" exitCode=0 Sep 10 19:24:46.205357 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.204947 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" event={"ID":"a4a21f16-293f-4a69-b781-2025c62bf661","Type":"ContainerDied","Data":"f5cfb0f1e59e6833c0669f125261dd6b289a2020335a9c88c5b1a576f86bf285"} Sep 10 19:24:46.205357 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.205014 2576 scope.go:117] "RemoveContainer" containerID="11bf4c423ac414b4c4b5bcdd6614d835e01a8bcc09ab702d633fd26f059f3f08" Sep 10 19:24:46.280699 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.280643 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:46.302746 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302718 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-istio-token\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.302873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302756 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-certs\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.302873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302792 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fktsh\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-kube-api-access-fktsh\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.302873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-data\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.302873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302862 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a21f16-293f-4a69-b781-2025c62bf661-istio-podinfo\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.303055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-credential-socket\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.303055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302916 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a21f16-293f-4a69-b781-2025c62bf661-istiod-ca-cert\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.303055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.302945 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-envoy\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.303055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303003 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-socket\") pod \"a4a21f16-293f-4a69-b781-2025c62bf661\" (UID: \"a4a21f16-293f-4a69-b781-2025c62bf661\") " Sep 10 19:24:46.303260 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303147 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:46.303260 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303172 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:46.303442 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303397 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.303442 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303432 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.303582 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303397 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:46.303582 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303452 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a21f16-293f-4a69-b781-2025c62bf661-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:24:46.303674 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.303553 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:46.306089 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.306050 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:24:46.306435 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.306395 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4a21f16-293f-4a69-b781-2025c62bf661-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:24:46.306541 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.306486 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:24:46.306541 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.306487 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-kube-api-access-fktsh" (OuterVolumeSpecName: "kube-api-access-fktsh") pod "a4a21f16-293f-4a69-b781-2025c62bf661" (UID: "a4a21f16-293f-4a69-b781-2025c62bf661"). InnerVolumeSpecName "kube-api-access-fktsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:24:46.404423 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.404393 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.404423 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.404421 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktsh\" (UniqueName: \"kubernetes.io/projected/a4a21f16-293f-4a69-b781-2025c62bf661-kube-api-access-fktsh\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.404423 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.404430 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a21f16-293f-4a69-b781-2025c62bf661-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.404423 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.404439 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.404689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.404451 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a21f16-293f-4a69-b781-2025c62bf661-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.404689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.404464 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:46.404689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:46.404471 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a21f16-293f-4a69-b781-2025c62bf661-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:24:47.212387 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:47.212345 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" Sep 10 19:24:47.212387 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:47.212363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w" event={"ID":"a4a21f16-293f-4a69-b781-2025c62bf661","Type":"ContainerDied","Data":"764479e306386e925d8048723cf3850b3e557b123cc9d68922e33486dea31553"} Sep 10 19:24:47.212907 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:47.212459 2576 scope.go:117] "RemoveContainer" containerID="f5cfb0f1e59e6833c0669f125261dd6b289a2020335a9c88c5b1a576f86bf285" Sep 10 19:24:47.249028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:47.249000 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w"] Sep 10 19:24:47.255877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:47.255853 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a888f7aa-openshift-default-7859c558b4-b882w"] Sep 10 19:24:48.602578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:48.602540 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" path="/var/lib/kubelet/pods/a4a21f16-293f-4a69-b781-2025c62bf661/volumes" Sep 10 19:24:49.976613 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:49.976573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:50.093931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:50.093899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:50.300663 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:50.300638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:50.464851 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:50.464812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:50.731356 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:50.731273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:24:56.612581 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:24:56.612552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:03.055828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:03.055797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:04.305858 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:04.305826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:05.158022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:05.157987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:10.983841 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:10.983806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:11.087311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.087273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:11.241184 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.241104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:11.684570 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.684528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:11.972401 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972301 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg"] Sep 10 19:25:11.972806 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972787 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" Sep 10 19:25:11.972902 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972808 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" Sep 10 19:25:11.972902 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972851 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e12cac-fb22-437a-8353-f4ce0cfc0f0b" containerName="revoke-keys" Sep 10 19:25:11.972902 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972860 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e12cac-fb22-437a-8353-f4ce0cfc0f0b" containerName="revoke-keys" Sep 10 19:25:11.973057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972958 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" Sep 10 19:25:11.973057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972971 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" Sep 10 19:25:11.973057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.972983 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e12cac-fb22-437a-8353-f4ce0cfc0f0b" containerName="revoke-keys" Sep 10 19:25:11.976278 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.976256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:11.979010 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.978991 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-717161-openshift-default-dockercfg-phscv\"" Sep 10 19:25:11.993254 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:11.993229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg"] Sep 10 19:25:12.013189 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.013160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:12.041542 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-socket\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.041690 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istiod-ca-cert\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.041690 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-credential-socket\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.041690 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgrh\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-kube-api-access-gqgrh\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.041690 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-certs\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.041888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-data\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.041888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-podinfo\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.041888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-token\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.042018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.041898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-envoy\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.121594 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.121554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:12.142613 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-envoy\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.142791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-socket\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.142791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istiod-ca-cert\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.142791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-credential-socket\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.142791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgrh\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-kube-api-access-gqgrh\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.142791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-certs\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.142791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-data\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.143098 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-podinfo\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.143098 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.142904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-token\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.143452 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.143423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-data\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.143452 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.143447 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istiod-ca-cert\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.143656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.143446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-certs\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.143775 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.143756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-socket\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.143844 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.143823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-credential-socket\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.145115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.145095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-envoy\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.145409 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.145388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-podinfo\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.151743 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.151722 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-token\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.152309 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.152278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgrh\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-kube-api-access-gqgrh\") pod \"e2e-explicit-717161-openshift-default-578d58f7b-ww6zg\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.287023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.286983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:12.576563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.576538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg"] Sep 10 19:25:12.577890 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:25:12.577862 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bcfbb6_1e96_4767_9bb0_f60636533d5b.slice/crio-a805cd0daf8dbadf2e877e2147ac402e8f14ffb336e1a66464acb3fbf81f8d1a WatchSource:0}: Error finding container a805cd0daf8dbadf2e877e2147ac402e8f14ffb336e1a66464acb3fbf81f8d1a: Status 404 returned error can't find the container with id a805cd0daf8dbadf2e877e2147ac402e8f14ffb336e1a66464acb3fbf81f8d1a Sep 10 19:25:12.583396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.580532 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:25:12.583396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.580699 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:25:12.583396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:12.580800 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:25:13.370506 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:13.370419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" event={"ID":"f9bcfbb6-1e96-4767-9bb0-f60636533d5b","Type":"ContainerStarted","Data":"168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb"} Sep 10 19:25:13.370506 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:13.370475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" event={"ID":"f9bcfbb6-1e96-4767-9bb0-f60636533d5b","Type":"ContainerStarted","Data":"a805cd0daf8dbadf2e877e2147ac402e8f14ffb336e1a66464acb3fbf81f8d1a"} Sep 10 19:25:13.392849 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:13.392800 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podStartSLOduration=2.392783013 podStartE2EDuration="2.392783013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:13.39157656 +0000 UTC m=+879.308232601" watchObservedRunningTime="2026-09-10 19:25:13.392783013 +0000 UTC m=+879.309439041" Sep 10 19:25:14.287706 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:14.287669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:14.292616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:14.292592 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:14.375602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:14.375571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:14.376580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:14.376563 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:15.330653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:15.330621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:15.428524 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:15.428496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:16.652237 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:16.652202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:16.749732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:16.749701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:16.968785 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:16.968696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:17.179660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:17.179629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:17.270069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:17.269982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:22.626701 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:22.626669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:22.699384 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:22.699351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:22.815037 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:22.815001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:22.913526 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:22.913435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:23.044597 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:23.044563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:26.557277 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:26.554092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:33.497749 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:33.497642 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerID="168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb" exitCode=0 Sep 10 19:25:33.497749 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:33.497720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" event={"ID":"f9bcfbb6-1e96-4767-9bb0-f60636533d5b","Type":"ContainerDied","Data":"168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb"} Sep 10 19:25:33.498187 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:33.498129 2576 scope.go:117] "RemoveContainer" containerID="168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb" Sep 10 19:25:33.499303 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:33.499265 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:25:33.499417 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:33.499354 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:25:33.499465 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:33.499416 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:25:34.505590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:34.505554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" event={"ID":"f9bcfbb6-1e96-4767-9bb0-f60636533d5b","Type":"ContainerStarted","Data":"84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b"} Sep 10 19:25:34.588878 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:34.588839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:25:34.589154 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:34.589138 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:25:35.287192 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:35.287153 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:35.287405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:35.287366 2576 patch_prober.go:28] interesting pod/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 19:25:35.287458 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:35.287437 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 19:25:36.288279 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:36.288243 2576 patch_prober.go:28] interesting pod/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 19:25:36.288708 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:36.288306 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 19:25:37.287556 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:37.287522 2576 patch_prober.go:28] interesting pod/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 19:25:37.287736 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:37.287574 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 19:25:38.288139 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:38.288103 2576 patch_prober.go:28] interesting pod/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 19:25:38.288583 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:38.288161 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 19:25:38.701861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:38.701780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:39.287572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:39.287533 2576 patch_prober.go:28] interesting pod/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 19:25:39.287757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:39.287597 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 19:25:40.289804 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:40.289769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:40.292543 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:40.292522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:40.292840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:40.292822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:40.293448 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:40.293430 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:25:44.585818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:44.585783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:44.699985 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:44.699954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:44.786554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:44.786521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:51.741879 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:51.741851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:55.281442 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:55.281367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:25:56.535120 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:25:56.535083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:02.188304 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:02.188274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:02.319931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:02.319900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:02.438741 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:02.438659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:02.536790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:02.536761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:03.160761 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:03.160734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:03.257714 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:03.257683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:03.360406 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:03.360360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:04.818957 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:04.818925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:05.056064 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:05.056033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:11.052232 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:11.052192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:11.211225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:11.211188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:11.727948 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:11.727886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:16.582540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.582502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:16.684324 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.684291 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw"] Sep 10 19:26:16.684733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.684718 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" Sep 10 19:26:16.684788 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.684734 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a21f16-293f-4a69-b781-2025c62bf661" containerName="istio-proxy" Sep 10 19:26:16.688030 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.688014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" Sep 10 19:26:16.690409 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.690355 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:26:16.690560 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.690518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:26:16.696983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.696954 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw"] Sep 10 19:26:16.760957 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.760919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2124c8-7f47-4176-9bae-6ae16d7a6c7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-717161-4sxgw\" (UID: \"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" Sep 10 19:26:16.861524 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.861441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2124c8-7f47-4176-9bae-6ae16d7a6c7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-717161-4sxgw\" (UID: \"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" Sep 10 19:26:16.862067 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:16.862047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2124c8-7f47-4176-9bae-6ae16d7a6c7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-717161-4sxgw\" (UID: \"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" Sep 10 19:26:17.001034 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:17.001004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" Sep 10 19:26:17.274912 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:17.274886 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw"] Sep 10 19:26:17.276204 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:26:17.276179 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2124c8_7f47_4176_9bae_6ae16d7a6c7f.slice/crio-24c7c755680de23a5ea72d787ae9e55e4756121fab266c5e1d913bf274fbe917 WatchSource:0}: Error finding container 24c7c755680de23a5ea72d787ae9e55e4756121fab266c5e1d913bf274fbe917: Status 404 returned error can't find the container with id 24c7c755680de23a5ea72d787ae9e55e4756121fab266c5e1d913bf274fbe917 Sep 10 19:26:17.762032 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:17.761996 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" event={"ID":"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f","Type":"ContainerStarted","Data":"95a68adbb20e97a08207b70506d146e886f763f500d23cbe6097151b55c1ec93"} Sep 10 19:26:17.762449 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:17.762049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" event={"ID":"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f","Type":"ContainerStarted","Data":"24c7c755680de23a5ea72d787ae9e55e4756121fab266c5e1d913bf274fbe917"} Sep 10 19:26:17.780589 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:17.780538 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" podStartSLOduration=1.7805236500000001 podStartE2EDuration="1.78052365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:17.777619491 +0000 UTC m=+943.694275521" watchObservedRunningTime="2026-09-10 19:26:17.78052365 +0000 UTC m=+943.697179679" Sep 10 19:26:18.070645 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:18.070615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:18.184909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:18.184876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:18.319008 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:18.318973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:18.612658 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:18.612575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:18.769498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:18.769338 2576 generic.go:356] "Generic (PLEG): container finished" podID="5c2124c8-7f47-4176-9bae-6ae16d7a6c7f" containerID="95a68adbb20e97a08207b70506d146e886f763f500d23cbe6097151b55c1ec93" exitCode=0 Sep 10 19:26:18.769498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:18.769426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" event={"ID":"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f","Type":"ContainerDied","Data":"95a68adbb20e97a08207b70506d146e886f763f500d23cbe6097151b55c1ec93"} Sep 10 19:26:18.885673 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:18.885591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:19.909118 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:19.909095 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" Sep 10 19:26:19.989224 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:19.989182 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2124c8-7f47-4176-9bae-6ae16d7a6c7f-maas-api-service-ca\") pod \"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f\" (UID: \"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f\") " Sep 10 19:26:19.989597 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:19.989573 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2124c8-7f47-4176-9bae-6ae16d7a6c7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c2124c8-7f47-4176-9bae-6ae16d7a6c7f" (UID: "5c2124c8-7f47-4176-9bae-6ae16d7a6c7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:26:20.089995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:20.089948 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2124c8-7f47-4176-9bae-6ae16d7a6c7f-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:20.782516 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:20.782484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" event={"ID":"5c2124c8-7f47-4176-9bae-6ae16d7a6c7f","Type":"ContainerDied","Data":"24c7c755680de23a5ea72d787ae9e55e4756121fab266c5e1d913bf274fbe917"} Sep 10 19:26:20.782682 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:20.782530 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw" Sep 10 19:26:20.782682 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:20.782536 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c7c755680de23a5ea72d787ae9e55e4756121fab266c5e1d913bf274fbe917" Sep 10 19:26:21.523077 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:21.523042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:21.607920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:21.607880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:22.833231 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:22.833169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:22.929553 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:22.929520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:23.113342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.113248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:23.366152 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.366058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:23.466655 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.466618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:23.498337 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.498301 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4"] Sep 10 19:26:23.498800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.498773 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2124c8-7f47-4176-9bae-6ae16d7a6c7f" containerName="revoke-keys" Sep 10 19:26:23.498800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.498794 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2124c8-7f47-4176-9bae-6ae16d7a6c7f" containerName="revoke-keys" Sep 10 19:26:23.498967 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.498935 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2124c8-7f47-4176-9bae-6ae16d7a6c7f" containerName="revoke-keys" Sep 10 19:26:23.502333 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.502313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:23.505295 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.505274 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c4487c09\"" Sep 10 19:26:23.517317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.517285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4"] Sep 10 19:26:23.621453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.621349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2ng\" (UniqueName: \"kubernetes.io/projected/c70d9190-c6e1-4553-b131-53dee6cafc4b-kube-api-access-2r2ng\") pod \"payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:23.621618 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.621541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d9190-c6e1-4553-b131-53dee6cafc4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:23.722318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.722278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2ng\" (UniqueName: \"kubernetes.io/projected/c70d9190-c6e1-4553-b131-53dee6cafc4b-kube-api-access-2r2ng\") pod \"payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:23.722524 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.722449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d9190-c6e1-4553-b131-53dee6cafc4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:23.723103 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.723079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d9190-c6e1-4553-b131-53dee6cafc4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:23.730954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.730926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2ng\" (UniqueName: \"kubernetes.io/projected/c70d9190-c6e1-4553-b131-53dee6cafc4b-kube-api-access-2r2ng\") pod \"payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:23.812534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:23.812504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:24.098800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:24.098775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4"] Sep 10 19:26:24.100877 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:26:24.100852 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70d9190_c6e1_4553_b131_53dee6cafc4b.slice/crio-783c2274c8c6e71681795c96ce7186e588300c0df08187e0b03e064ff34fb7ca WatchSource:0}: Error finding container 783c2274c8c6e71681795c96ce7186e588300c0df08187e0b03e064ff34fb7ca: Status 404 returned error can't find the container with id 783c2274c8c6e71681795c96ce7186e588300c0df08187e0b03e064ff34fb7ca Sep 10 19:26:24.809018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:24.808983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" event={"ID":"c70d9190-c6e1-4553-b131-53dee6cafc4b","Type":"ContainerStarted","Data":"19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980"} Sep 10 19:26:24.809209 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:24.809036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" event={"ID":"c70d9190-c6e1-4553-b131-53dee6cafc4b","Type":"ContainerStarted","Data":"783c2274c8c6e71681795c96ce7186e588300c0df08187e0b03e064ff34fb7ca"} Sep 10 19:26:24.809209 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:24.809059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:24.829644 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:24.829598 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" podStartSLOduration=1.829583842 podStartE2EDuration="1.829583842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:24.829475309 +0000 UTC m=+950.746131337" watchObservedRunningTime="2026-09-10 19:26:24.829583842 +0000 UTC m=+950.746239870" Sep 10 19:26:26.949539 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:26.949504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:27.626756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:27.626716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:27.702652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:27.702623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:27.769611 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:27.769578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:27.840179 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:27.840145 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw"] Sep 10 19:26:27.844846 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:27.844820 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-717161-4sxgw"] Sep 10 19:26:27.865678 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:27.865638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:28.212597 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:28.212556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:28.607560 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:28.607522 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c2124c8-7f47-4176-9bae-6ae16d7a6c7f" path="/var/lib/kubelet/pods/5c2124c8-7f47-4176-9bae-6ae16d7a6c7f/volumes" Sep 10 19:26:33.280023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:33.279986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:35.022616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:35.022586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:35.192839 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:35.192802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:35.270493 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:35.270465 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg"] Sep 10 19:26:35.270715 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:35.270694 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" containerID="cri-o://84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b" gracePeriod=30 Sep 10 19:26:35.286113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:35.286091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:35.706432 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:35.706337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:35.817886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:35.817858 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:26:36.000315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:36.000229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:36.101443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:36.101358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:37.012610 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:37.012576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:37.077477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:37.077449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:37.185764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:37.185733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:40.294807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.294776 2576 patch_prober.go:28] interesting pod/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 19:26:40.295103 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.294845 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 19:26:40.530086 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.530063 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:26:40.682755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.682721 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-podinfo\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.682951 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.682789 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-socket\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.682951 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.682838 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqgrh\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-kube-api-access-gqgrh\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.682951 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.682898 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-data\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.682951 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.682926 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-envoy\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.683169 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.682997 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-certs\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.683169 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683029 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-credential-socket\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.683169 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683054 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istiod-ca-cert\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.683169 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683044 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:26:40.683169 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683087 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-token\") pod \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\" (UID: \"f9bcfbb6-1e96-4767-9bb0-f60636533d5b\") " Sep 10 19:26:40.683471 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683232 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:26:40.683471 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683303 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:26:40.683471 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683414 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:26:40.683660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683509 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.683660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683524 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.683660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683521 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:26:40.683660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683537 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.683660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.683549 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.685293 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.685258 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:26:40.685293 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.685276 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:26:40.685617 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.685591 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-kube-api-access-gqgrh" (OuterVolumeSpecName: "kube-api-access-gqgrh") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "kube-api-access-gqgrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:26:40.685709 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.685613 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9bcfbb6-1e96-4767-9bb0-f60636533d5b" (UID: "f9bcfbb6-1e96-4767-9bb0-f60636533d5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:26:40.784268 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.784234 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.784268 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.784262 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgrh\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-kube-api-access-gqgrh\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.784268 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.784272 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.784530 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.784280 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.784530 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.784288 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcfbb6-1e96-4767-9bb0-f60636533d5b-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:40.911065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.911033 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerID="84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b" exitCode=0 Sep 10 19:26:40.911215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.911081 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" Sep 10 19:26:40.911215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.911118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" event={"ID":"f9bcfbb6-1e96-4767-9bb0-f60636533d5b","Type":"ContainerDied","Data":"84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b"} Sep 10 19:26:40.911215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.911176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg" event={"ID":"f9bcfbb6-1e96-4767-9bb0-f60636533d5b","Type":"ContainerDied","Data":"a805cd0daf8dbadf2e877e2147ac402e8f14ffb336e1a66464acb3fbf81f8d1a"} Sep 10 19:26:40.911215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.911207 2576 scope.go:117] "RemoveContainer" containerID="84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b" Sep 10 19:26:40.921641 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.921616 2576 scope.go:117] "RemoveContainer" containerID="168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb" Sep 10 19:26:40.930014 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.929996 2576 scope.go:117] "RemoveContainer" containerID="84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b" Sep 10 19:26:40.930286 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:26:40.930268 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b\": container with ID starting with 84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b not found: ID does not exist" containerID="84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b" Sep 10 19:26:40.930360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.930299 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b"} err="failed to get container status \"84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b\": rpc error: code = NotFound desc = could not find container \"84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b\": container with ID starting with 84efd24a31ed0e2bb5088f9603417ce1550865c012fc348384180d0fad2c147b not found: ID does not exist" Sep 10 19:26:40.930360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.930322 2576 scope.go:117] "RemoveContainer" containerID="168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb" Sep 10 19:26:40.930609 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:26:40.930590 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb\": container with ID starting with 168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb not found: ID does not exist" containerID="168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb" Sep 10 19:26:40.930653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.930615 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb"} err="failed to get container status \"168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb\": rpc error: code = NotFound desc = could not find container \"168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb\": container with ID starting with 168dc678f849a86dd13372a6a99cbb84ef2a3fba290a8cbdbde34c43cdd806cb not found: ID does not exist" Sep 10 19:26:40.943017 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.942958 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg"] Sep 10 19:26:40.950790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:40.950766 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-717161-openshift-default-578d58f7b-ww6zg"] Sep 10 19:26:42.602591 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:42.602562 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" path="/var/lib/kubelet/pods/f9bcfbb6-1e96-4767-9bb0-f60636533d5b/volumes" Sep 10 19:26:43.832160 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:43.832128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:46.502948 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:46.502915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:47.261924 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:47.261884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:47.382306 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:47.382264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:47.670492 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:47.670463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:49.231501 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:49.231462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:54.113426 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.113396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:54.209482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.209446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:54.444795 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.444718 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf"] Sep 10 19:26:54.445138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.445123 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" Sep 10 19:26:54.445187 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.445141 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" Sep 10 19:26:54.445235 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.445225 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" Sep 10 19:26:54.445271 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.445239 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" Sep 10 19:26:54.450045 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.450027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" Sep 10 19:26:54.453063 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.453035 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:26:54.453211 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.453186 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:26:54.458855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.458831 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf"] Sep 10 19:26:54.513767 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.513732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf\" (UID: \"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" Sep 10 19:26:54.614270 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.614235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf\" (UID: \"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" Sep 10 19:26:54.614863 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.614839 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf\" (UID: \"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" Sep 10 19:26:54.763542 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:54.763464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" Sep 10 19:26:55.039919 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:55.039895 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf"] Sep 10 19:26:55.041819 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:26:55.041789 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fefa2d_bf47_4b0d_b051_25bb8bfbf3fd.slice/crio-d8b784b811ff4fd8f3153fafb4d585579d72c59e2d33e7fedb59892980047ae1 WatchSource:0}: Error finding container d8b784b811ff4fd8f3153fafb4d585579d72c59e2d33e7fedb59892980047ae1: Status 404 returned error can't find the container with id d8b784b811ff4fd8f3153fafb4d585579d72c59e2d33e7fedb59892980047ae1 Sep 10 19:26:55.998168 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:55.998052 2576 generic.go:356] "Generic (PLEG): container finished" podID="77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd" containerID="45935b06eba9e0d7282a6356090c66de8f45627d5d68628ce703ba69c5411cdc" exitCode=0 Sep 10 19:26:55.998168 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:55.998130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" event={"ID":"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd","Type":"ContainerDied","Data":"45935b06eba9e0d7282a6356090c66de8f45627d5d68628ce703ba69c5411cdc"} Sep 10 19:26:55.998579 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:55.998180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" event={"ID":"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd","Type":"ContainerStarted","Data":"d8b784b811ff4fd8f3153fafb4d585579d72c59e2d33e7fedb59892980047ae1"} Sep 10 19:26:56.854225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:56.854190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:56.969162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:56.969134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:57.147396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:57.147346 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" Sep 10 19:26:57.180997 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:57.180966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:57.234551 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:57.234520 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd-maas-api-service-ca\") pod \"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd\" (UID: \"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd\") " Sep 10 19:26:57.234880 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:57.234855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd" (UID: "77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:26:57.289963 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:57.289932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:26:57.335124 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:57.335091 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:26:58.010707 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:58.010675 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" Sep 10 19:26:58.010919 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:58.010715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf" event={"ID":"77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd","Type":"ContainerDied","Data":"d8b784b811ff4fd8f3153fafb4d585579d72c59e2d33e7fedb59892980047ae1"} Sep 10 19:26:58.010919 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:26:58.010760 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b784b811ff4fd8f3153fafb4d585579d72c59e2d33e7fedb59892980047ae1" Sep 10 19:27:02.322387 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:02.322348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:04.985646 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:04.985610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:05.012213 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.012181 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4"] Sep 10 19:27:05.012510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.012480 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" podUID="c70d9190-c6e1-4553-b131-53dee6cafc4b" containerName="payload-processing" containerID="cri-o://19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980" gracePeriod=30 Sep 10 19:27:05.087657 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.087623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:05.188034 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.188002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:05.196800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.196775 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf"] Sep 10 19:27:05.200989 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.200960 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4487c09-d7jxf"] Sep 10 19:27:05.259535 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.259511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:05.275629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.275607 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:27:05.308120 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.308001 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d9190-c6e1-4553-b131-53dee6cafc4b-plugins-config-volume\") pod \"c70d9190-c6e1-4553-b131-53dee6cafc4b\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " Sep 10 19:27:05.308320 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.308202 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r2ng\" (UniqueName: \"kubernetes.io/projected/c70d9190-c6e1-4553-b131-53dee6cafc4b-kube-api-access-2r2ng\") pod \"c70d9190-c6e1-4553-b131-53dee6cafc4b\" (UID: \"c70d9190-c6e1-4553-b131-53dee6cafc4b\") " Sep 10 19:27:05.308462 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.308440 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70d9190-c6e1-4553-b131-53dee6cafc4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c70d9190-c6e1-4553-b131-53dee6cafc4b" (UID: "c70d9190-c6e1-4553-b131-53dee6cafc4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:05.308728 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.308705 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d9190-c6e1-4553-b131-53dee6cafc4b-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:05.310796 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.310767 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70d9190-c6e1-4553-b131-53dee6cafc4b-kube-api-access-2r2ng" (OuterVolumeSpecName: "kube-api-access-2r2ng") pod "c70d9190-c6e1-4553-b131-53dee6cafc4b" (UID: "c70d9190-c6e1-4553-b131-53dee6cafc4b"). InnerVolumeSpecName "kube-api-access-2r2ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:27:05.409697 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.409652 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r2ng\" (UniqueName: \"kubernetes.io/projected/c70d9190-c6e1-4553-b131-53dee6cafc4b-kube-api-access-2r2ng\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:05.580620 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:05.580585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:06.066829 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.066789 2576 generic.go:356] "Generic (PLEG): container finished" podID="c70d9190-c6e1-4553-b131-53dee6cafc4b" containerID="19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980" exitCode=0 Sep 10 19:27:06.067314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.066827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" event={"ID":"c70d9190-c6e1-4553-b131-53dee6cafc4b","Type":"ContainerDied","Data":"19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980"} Sep 10 19:27:06.067314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.066847 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" Sep 10 19:27:06.067314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.066932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4" event={"ID":"c70d9190-c6e1-4553-b131-53dee6cafc4b","Type":"ContainerDied","Data":"783c2274c8c6e71681795c96ce7186e588300c0df08187e0b03e064ff34fb7ca"} Sep 10 19:27:06.067314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.066968 2576 scope.go:117] "RemoveContainer" containerID="19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980" Sep 10 19:27:06.076533 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.076515 2576 scope.go:117] "RemoveContainer" containerID="19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980" Sep 10 19:27:06.076834 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:27:06.076814 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980\": container with ID starting with 19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980 not found: ID does not exist" containerID="19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980" Sep 10 19:27:06.076887 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.076846 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980"} err="failed to get container status \"19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980\": rpc error: code = NotFound desc = could not find container \"19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980\": container with ID starting with 19f7afbbf09d533d2b7cb42c5550cd217c851c6ea233bca385139c1cea71c980 not found: ID does not exist" Sep 10 19:27:06.089785 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.089757 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4"] Sep 10 19:27:06.096412 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.096389 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4487c09-647bd59c69-fh6d4"] Sep 10 19:27:06.602807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.602775 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd" path="/var/lib/kubelet/pods/77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd/volumes" Sep 10 19:27:06.603140 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:06.603128 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c70d9190-c6e1-4553-b131-53dee6cafc4b" path="/var/lib/kubelet/pods/c70d9190-c6e1-4553-b131-53dee6cafc4b/volumes" Sep 10 19:27:09.879143 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:09.879111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:10.964654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:10.964619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:11.330595 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:11.330546 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:12.206965 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:12.206921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:12.309264 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:12.309229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:12.565549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:12.565516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:12.879996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:12.879904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:12.978495 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:12.978467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:13.074474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:13.074444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:19.655656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:19.655628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:19.857433 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:19.857403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:19.988695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:19.988602 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:21.845389 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:21.845350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:21.878076 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:21.878044 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6"] Sep 10 19:27:21.878278 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:21.878254 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" podUID="6917793d-c854-45ac-8ccf-8e837fc97110" containerName="payload-processing" containerID="cri-o://d490621ab7b24eb076ebc24a0339a9712998e56eaba272af4b42381353268c04" gracePeriod=30 Sep 10 19:27:21.889122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:21.889094 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn"] Sep 10 19:27:21.889875 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:21.889310 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" podUID="82b1b9cc-8ca7-449d-be53-c0fff0d51a22" containerName="payload-pre-processing" containerID="cri-o://85e3de1d82c8e83a0afa6067cb61195d8234a2625d9a6a4a717e84a988a79bc7" gracePeriod=30 Sep 10 19:27:21.942904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:21.942878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:22.027391 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.027340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:22.114760 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.114726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:22.166292 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.166259 2576 generic.go:356] "Generic (PLEG): container finished" podID="82b1b9cc-8ca7-449d-be53-c0fff0d51a22" containerID="85e3de1d82c8e83a0afa6067cb61195d8234a2625d9a6a4a717e84a988a79bc7" exitCode=0 Sep 10 19:27:22.166427 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.166335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" event={"ID":"82b1b9cc-8ca7-449d-be53-c0fff0d51a22","Type":"ContainerDied","Data":"85e3de1d82c8e83a0afa6067cb61195d8234a2625d9a6a4a717e84a988a79bc7"} Sep 10 19:27:22.166427 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.166410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" event={"ID":"82b1b9cc-8ca7-449d-be53-c0fff0d51a22","Type":"ContainerDied","Data":"a08dd8bf5cfa39f1ef31a7478b778111ff536c6aaabbeb752179f202d414518d"} Sep 10 19:27:22.166528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.166435 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a08dd8bf5cfa39f1ef31a7478b778111ff536c6aaabbeb752179f202d414518d" Sep 10 19:27:22.167895 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.167811 2576 generic.go:356] "Generic (PLEG): container finished" podID="6917793d-c854-45ac-8ccf-8e837fc97110" containerID="d490621ab7b24eb076ebc24a0339a9712998e56eaba272af4b42381353268c04" exitCode=0 Sep 10 19:27:22.167895 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.167861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" event={"ID":"6917793d-c854-45ac-8ccf-8e837fc97110","Type":"ContainerDied","Data":"d490621ab7b24eb076ebc24a0339a9712998e56eaba272af4b42381353268c04"} Sep 10 19:27:22.167895 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.167893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" event={"ID":"6917793d-c854-45ac-8ccf-8e837fc97110","Type":"ContainerDied","Data":"fd92b4bec1a82409fb26640f16b67fee14aecb2b0d2243c7a4aadd25cefbea9d"} Sep 10 19:27:22.168051 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.167907 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd92b4bec1a82409fb26640f16b67fee14aecb2b0d2243c7a4aadd25cefbea9d" Sep 10 19:27:22.177616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.177599 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:27:22.180702 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.180675 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:27:22.267293 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.267268 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swzhb\" (UniqueName: \"kubernetes.io/projected/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-kube-api-access-swzhb\") pod \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " Sep 10 19:27:22.267510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.267308 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-plugins-config-volume\") pod \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\" (UID: \"82b1b9cc-8ca7-449d-be53-c0fff0d51a22\") " Sep 10 19:27:22.267510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.267333 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmg5s\" (UniqueName: \"kubernetes.io/projected/6917793d-c854-45ac-8ccf-8e837fc97110-kube-api-access-mmg5s\") pod \"6917793d-c854-45ac-8ccf-8e837fc97110\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " Sep 10 19:27:22.267510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.267360 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6917793d-c854-45ac-8ccf-8e837fc97110-plugins-config-volume\") pod \"6917793d-c854-45ac-8ccf-8e837fc97110\" (UID: \"6917793d-c854-45ac-8ccf-8e837fc97110\") " Sep 10 19:27:22.267764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.267741 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82b1b9cc-8ca7-449d-be53-c0fff0d51a22" (UID: "82b1b9cc-8ca7-449d-be53-c0fff0d51a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:22.267813 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.267766 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6917793d-c854-45ac-8ccf-8e837fc97110-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6917793d-c854-45ac-8ccf-8e837fc97110" (UID: "6917793d-c854-45ac-8ccf-8e837fc97110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:22.269655 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.269628 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-kube-api-access-swzhb" (OuterVolumeSpecName: "kube-api-access-swzhb") pod "82b1b9cc-8ca7-449d-be53-c0fff0d51a22" (UID: "82b1b9cc-8ca7-449d-be53-c0fff0d51a22"). InnerVolumeSpecName "kube-api-access-swzhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:27:22.269796 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.269775 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6917793d-c854-45ac-8ccf-8e837fc97110-kube-api-access-mmg5s" (OuterVolumeSpecName: "kube-api-access-mmg5s") pod "6917793d-c854-45ac-8ccf-8e837fc97110" (UID: "6917793d-c854-45ac-8ccf-8e837fc97110"). InnerVolumeSpecName "kube-api-access-mmg5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:27:22.369047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.368957 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6917793d-c854-45ac-8ccf-8e837fc97110-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:22.369047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.368987 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzhb\" (UniqueName: \"kubernetes.io/projected/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-kube-api-access-swzhb\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:22.369047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.368999 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b1b9cc-8ca7-449d-be53-c0fff0d51a22-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:22.369047 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.369011 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmg5s\" (UniqueName: \"kubernetes.io/projected/6917793d-c854-45ac-8ccf-8e837fc97110-kube-api-access-mmg5s\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:22.476794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:22.476758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:23.173796 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:23.173762 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn" Sep 10 19:27:23.174196 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:23.173762 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6" Sep 10 19:27:23.203978 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:23.203950 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6"] Sep 10 19:27:23.210739 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:23.210704 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b562c-7b54fb8ff7-gqcf6"] Sep 10 19:27:23.226914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:23.226889 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn"] Sep 10 19:27:23.233757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:23.233730 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b562c-7cb5b5594-t64qn"] Sep 10 19:27:24.601951 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:24.601917 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6917793d-c854-45ac-8ccf-8e837fc97110" path="/var/lib/kubelet/pods/6917793d-c854-45ac-8ccf-8e837fc97110/volumes" Sep 10 19:27:24.602315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:24.602252 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b1b9cc-8ca7-449d-be53-c0fff0d51a22" path="/var/lib/kubelet/pods/82b1b9cc-8ca7-449d-be53-c0fff0d51a22/volumes" Sep 10 19:27:26.185927 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:26.185896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:26.270573 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:26.270537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:26.701426 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:26.701366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:27.108111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:27.108079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:27.429751 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:27.429678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:27.617996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:27.617955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:30.024258 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:30.024219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:30.120541 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:30.120501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:30.307561 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:30.307526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:30.740775 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:30.740687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:30.989149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:30.989120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:31.128772 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:31.128707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:32.675741 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:32.675705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:32.778323 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:32.778280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:32.887901 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:32.887872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:33.208391 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.208329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:33.304632 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.304597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:33.400663 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.400627 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj"] Sep 10 19:27:33.401085 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401067 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6917793d-c854-45ac-8ccf-8e837fc97110" containerName="payload-processing" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401089 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6917793d-c854-45ac-8ccf-8e837fc97110" containerName="payload-processing" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401104 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b1b9cc-8ca7-449d-be53-c0fff0d51a22" containerName="payload-pre-processing" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401112 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b1b9cc-8ca7-449d-be53-c0fff0d51a22" containerName="payload-pre-processing" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401127 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70d9190-c6e1-4553-b131-53dee6cafc4b" containerName="payload-processing" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401136 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70d9190-c6e1-4553-b131-53dee6cafc4b" containerName="payload-processing" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401157 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401165 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9bcfbb6-1e96-4767-9bb0-f60636533d5b" containerName="istio-proxy" Sep 10 19:27:33.401174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401173 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd" containerName="revoke-keys" Sep 10 19:27:33.401605 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401182 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd" containerName="revoke-keys" Sep 10 19:27:33.401605 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401272 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6917793d-c854-45ac-8ccf-8e837fc97110" containerName="payload-processing" Sep 10 19:27:33.401605 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401285 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b1b9cc-8ca7-449d-be53-c0fff0d51a22" containerName="payload-pre-processing" Sep 10 19:27:33.401605 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401297 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70d9190-c6e1-4553-b131-53dee6cafc4b" containerName="payload-processing" Sep 10 19:27:33.401605 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.401308 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77fefa2d-bf47-4b0d-b051-25bb8bfbf3fd" containerName="revoke-keys" Sep 10 19:27:33.404596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.404575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.408445 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.408424 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cfa578\"" Sep 10 19:27:33.408565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.408424 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cfa578-dockercfg-zdhbj\"" Sep 10 19:27:33.413082 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.413057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj"] Sep 10 19:27:33.570976 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.570942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592c00f4-ce09-44f5-be90-42c236292006-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.571128 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.570996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2d2q\" (UniqueName: \"kubernetes.io/projected/592c00f4-ce09-44f5-be90-42c236292006-kube-api-access-m2d2q\") pod \"payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.672388 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.672346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592c00f4-ce09-44f5-be90-42c236292006-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.672566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.672418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2d2q\" (UniqueName: \"kubernetes.io/projected/592c00f4-ce09-44f5-be90-42c236292006-kube-api-access-m2d2q\") pod \"payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.673122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.673101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592c00f4-ce09-44f5-be90-42c236292006-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.689653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.689630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2d2q\" (UniqueName: \"kubernetes.io/projected/592c00f4-ce09-44f5-be90-42c236292006-kube-api-access-m2d2q\") pod \"payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.717495 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.717471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:33.990886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:33.990857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj"] Sep 10 19:27:33.992158 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:27:33.992124 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592c00f4_ce09_44f5_be90_42c236292006.slice/crio-c55f993cc8cb11e5bb093f6154763790b18f6baf0f798026090482b5d2f9f463 WatchSource:0}: Error finding container c55f993cc8cb11e5bb093f6154763790b18f6baf0f798026090482b5d2f9f463: Status 404 returned error can't find the container with id c55f993cc8cb11e5bb093f6154763790b18f6baf0f798026090482b5d2f9f463 Sep 10 19:27:34.242512 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:34.242407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" event={"ID":"592c00f4-ce09-44f5-be90-42c236292006","Type":"ContainerStarted","Data":"6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49"} Sep 10 19:27:34.242512 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:34.242482 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:34.242755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:34.242512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" event={"ID":"592c00f4-ce09-44f5-be90-42c236292006","Type":"ContainerStarted","Data":"c55f993cc8cb11e5bb093f6154763790b18f6baf0f798026090482b5d2f9f463"} Sep 10 19:27:34.268754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:34.268710 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" podStartSLOduration=1.268691185 podStartE2EDuration="1.268691185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:34.268032046 +0000 UTC m=+1020.184688077" watchObservedRunningTime="2026-09-10 19:27:34.268691185 +0000 UTC m=+1020.185347191" Sep 10 19:27:39.443708 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:39.443674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:40.901021 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:40.900981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:41.840575 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:41.840547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:41.879724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:41.879687 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt"] Sep 10 19:27:41.880033 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:41.879989 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" podUID="4e5f41c6-4458-492d-8943-2dff54201505" containerName="payload-processing" containerID="cri-o://71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca" gracePeriod=30 Sep 10 19:27:41.986202 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:41.986167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:42.111594 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.111459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:42.152578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.152551 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:27:42.249839 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.249809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:42.290070 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.290037 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e5f41c6-4458-492d-8943-2dff54201505" containerID="71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca" exitCode=0 Sep 10 19:27:42.290267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.290081 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" Sep 10 19:27:42.290267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.290116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" event={"ID":"4e5f41c6-4458-492d-8943-2dff54201505","Type":"ContainerDied","Data":"71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca"} Sep 10 19:27:42.290267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.290172 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt" event={"ID":"4e5f41c6-4458-492d-8943-2dff54201505","Type":"ContainerDied","Data":"b7720296a54723e7573fcfc5844dbd5df667570ea709f0d9d575e02b259c5713"} Sep 10 19:27:42.290267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.290201 2576 scope.go:117] "RemoveContainer" containerID="71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca" Sep 10 19:27:42.299192 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.299176 2576 scope.go:117] "RemoveContainer" containerID="71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca" Sep 10 19:27:42.299470 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:27:42.299447 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca\": container with ID starting with 71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca not found: ID does not exist" containerID="71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca" Sep 10 19:27:42.299554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.299483 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca"} err="failed to get container status \"71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca\": rpc error: code = NotFound desc = could not find container \"71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca\": container with ID starting with 71705b5bc6f8a6fc6efc8c200357b93012584f41f89c6d00d7e5735228e7f7ca not found: ID does not exist" Sep 10 19:27:42.351825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.351797 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkdkt\" (UniqueName: \"kubernetes.io/projected/4e5f41c6-4458-492d-8943-2dff54201505-kube-api-access-dkdkt\") pod \"4e5f41c6-4458-492d-8943-2dff54201505\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " Sep 10 19:27:42.351942 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.351830 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f41c6-4458-492d-8943-2dff54201505-plugins-config-volume\") pod \"4e5f41c6-4458-492d-8943-2dff54201505\" (UID: \"4e5f41c6-4458-492d-8943-2dff54201505\") " Sep 10 19:27:42.352188 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.352168 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5f41c6-4458-492d-8943-2dff54201505-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e5f41c6-4458-492d-8943-2dff54201505" (UID: "4e5f41c6-4458-492d-8943-2dff54201505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:42.353991 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.353966 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5f41c6-4458-492d-8943-2dff54201505-kube-api-access-dkdkt" (OuterVolumeSpecName: "kube-api-access-dkdkt") pod "4e5f41c6-4458-492d-8943-2dff54201505" (UID: "4e5f41c6-4458-492d-8943-2dff54201505"). InnerVolumeSpecName "kube-api-access-dkdkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:27:42.452648 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.452569 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkdkt\" (UniqueName: \"kubernetes.io/projected/4e5f41c6-4458-492d-8943-2dff54201505-kube-api-access-dkdkt\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:42.452648 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.452598 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f41c6-4458-492d-8943-2dff54201505-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:42.620177 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.620144 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt"] Sep 10 19:27:42.624345 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.624317 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b960a2-7b779975c9-rrtqt"] Sep 10 19:27:42.628908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:42.628882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:44.602709 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:44.602673 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5f41c6-4458-492d-8943-2dff54201505" path="/var/lib/kubelet/pods/4e5f41c6-4458-492d-8943-2dff54201505/volumes" Sep 10 19:27:45.251174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:45.251148 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:27:48.955538 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:48.955505 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:49.053628 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:49.053595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:49.188052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:49.188022 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt"] Sep 10 19:27:49.188295 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:49.188249 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" containerID="cri-o://fd85a53eef4ae019e14c228521087e175c3f70ef0fe0e4ad56ef86c73511b785" gracePeriod=30 Sep 10 19:27:49.218894 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:49.218811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:49.815646 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:49.815618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:50.082867 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.082789 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l"] Sep 10 19:27:50.083228 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.083175 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5f41c6-4458-492d-8943-2dff54201505" containerName="payload-processing" Sep 10 19:27:50.083228 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.083191 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5f41c6-4458-492d-8943-2dff54201505" containerName="payload-processing" Sep 10 19:27:50.083315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.083261 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5f41c6-4458-492d-8943-2dff54201505" containerName="payload-processing" Sep 10 19:27:50.087788 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.087764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.092390 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.092355 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b64b1c-openshift-default-dockercfg-ggv4c\"" Sep 10 19:27:50.115693 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbq45\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-kube-api-access-qbq45\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.115814 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-certs\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.115814 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-data\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.115907 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01845e69-4217-4e5e-b6b7-c255f3325067-istio-podinfo\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.115907 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-envoy\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.115907 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01845e69-4217-4e5e-b6b7-c255f3325067-istiod-ca-cert\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.116045 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-socket\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.116045 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.115950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-istio-token\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.116045 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.116015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-credential-socket\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.116864 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.116844 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l"] Sep 10 19:27:50.142003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.141959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:50.216537 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-socket\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-istio-token\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-credential-socket\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbq45\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-kube-api-access-qbq45\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-certs\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216961 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-data\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216961 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01845e69-4217-4e5e-b6b7-c255f3325067-istio-podinfo\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216961 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-envoy\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.216961 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01845e69-4217-4e5e-b6b7-c255f3325067-istiod-ca-cert\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.217147 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-socket\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.217147 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.216992 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-certs\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.217147 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.217045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-credential-socket\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.217249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.217167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-data\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.217600 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.217567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01845e69-4217-4e5e-b6b7-c255f3325067-istiod-ca-cert\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.219267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.219246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-envoy\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.219412 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.219396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01845e69-4217-4e5e-b6b7-c255f3325067-istio-podinfo\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.232177 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.232148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-istio-token\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.232383 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.232352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbq45\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-kube-api-access-qbq45\") pod \"e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.398230 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.398146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:50.678057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.678034 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l"] Sep 10 19:27:50.679288 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:27:50.679261 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01845e69_4217_4e5e_b6b7_c255f3325067.slice/crio-b57038c4b4edf304864bf0e0a6c9c20c936304e05a9597a8a1dee893b44590a1 WatchSource:0}: Error finding container b57038c4b4edf304864bf0e0a6c9c20c936304e05a9597a8a1dee893b44590a1: Status 404 returned error can't find the container with id b57038c4b4edf304864bf0e0a6c9c20c936304e05a9597a8a1dee893b44590a1 Sep 10 19:27:50.681492 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.681454 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:27:50.681585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.681525 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:27:50.681585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.681579 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:27:50.809888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:50.809856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:51.346776 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:51.346737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" event={"ID":"01845e69-4217-4e5e-b6b7-c255f3325067","Type":"ContainerStarted","Data":"55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5"} Sep 10 19:27:51.347187 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:51.346798 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" event={"ID":"01845e69-4217-4e5e-b6b7-c255f3325067","Type":"ContainerStarted","Data":"b57038c4b4edf304864bf0e0a6c9c20c936304e05a9597a8a1dee893b44590a1"} Sep 10 19:27:51.370023 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:51.369982 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podStartSLOduration=1.369967762 podStartE2EDuration="1.369967762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:51.368732856 +0000 UTC m=+1037.285388886" watchObservedRunningTime="2026-09-10 19:27:51.369967762 +0000 UTC m=+1037.286623791" Sep 10 19:27:51.398750 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:51.398721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:51.404221 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:51.404199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:52.352818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:52.352782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:52.353858 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:52.353844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:27:53.315709 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:53.315674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:53.450986 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:53.450953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:53.689671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:53.689591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:54.369037 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.369008 2576 generic.go:356] "Generic (PLEG): container finished" podID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerID="fd85a53eef4ae019e14c228521087e175c3f70ef0fe0e4ad56ef86c73511b785" exitCode=0 Sep 10 19:27:54.369199 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.369079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" event={"ID":"6f5b1544-0e93-4aba-b06c-900c311b3db8","Type":"ContainerDied","Data":"fd85a53eef4ae019e14c228521087e175c3f70ef0fe0e4ad56ef86c73511b785"} Sep 10 19:27:54.369199 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.369153 2576 scope.go:117] "RemoveContainer" containerID="1e6789eec89b9e6e05995573052425ba051f79c9f8760a2c9a6298383dedf21b" Sep 10 19:27:54.451689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.451618 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:27:54.557631 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557602 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-socket\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.557800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557660 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-podinfo\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.557800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-credential-socket\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.557800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-certs\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.557800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557733 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-token\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.557800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557778 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xtj9\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-kube-api-access-8xtj9\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5b1544-0e93-4aba-b06c-900c311b3db8-istiod-ca-cert\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557876 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-envoy\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557895 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.557905 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-data\") pod \"6f5b1544-0e93-4aba-b06c-900c311b3db8\" (UID: \"6f5b1544-0e93-4aba-b06c-900c311b3db8\") " Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.558009 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.558038 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.558311 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.558331 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.558347 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.558344 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5b1544-0e93-4aba-b06c-900c311b3db8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:27:54.558662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.558585 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:27:54.560177 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.560148 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:27:54.560714 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.560677 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:27:54.560807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.560718 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-kube-api-access-8xtj9" (OuterVolumeSpecName: "kube-api-access-8xtj9") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "kube-api-access-8xtj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:27:54.560807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.560769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f5b1544-0e93-4aba-b06c-900c311b3db8" (UID: "6f5b1544-0e93-4aba-b06c-900c311b3db8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:27:54.659578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.659542 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5b1544-0e93-4aba-b06c-900c311b3db8-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.659578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.659572 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.659578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.659583 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.659817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.659597 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.659817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.659607 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.659817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.659619 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xtj9\" (UniqueName: \"kubernetes.io/projected/6f5b1544-0e93-4aba-b06c-900c311b3db8-kube-api-access-8xtj9\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:27:54.846571 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.846526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:54.938917 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:54.938879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:55.111599 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.111512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:55.376243 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.376167 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" Sep 10 19:27:55.376243 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.376207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt" event={"ID":"6f5b1544-0e93-4aba-b06c-900c311b3db8","Type":"ContainerDied","Data":"0a75a14b83cd4e001877043cda1cf76167c8d32cacc610299eb24c46e23f9696"} Sep 10 19:27:55.376507 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.376264 2576 scope.go:117] "RemoveContainer" containerID="fd85a53eef4ae019e14c228521087e175c3f70ef0fe0e4ad56ef86c73511b785" Sep 10 19:27:55.404643 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.404614 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt"] Sep 10 19:27:55.416314 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.416285 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b960a2-openshift-default-84bf98d484-d68kt"] Sep 10 19:27:55.593848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.593815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:55.683112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.683031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:55.707896 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.707866 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z"] Sep 10 19:27:55.708228 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.708215 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" Sep 10 19:27:55.708269 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.708230 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" Sep 10 19:27:55.708326 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.708318 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" Sep 10 19:27:55.708364 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.708328 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" Sep 10 19:27:55.712748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.712729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:55.715410 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.715391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b64b1c\"" Sep 10 19:27:55.715504 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.715453 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b64b1c-dockercfg-j7vpm\"" Sep 10 19:27:55.724771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.724748 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z"] Sep 10 19:27:55.770265 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.770229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbpr\" (UniqueName: \"kubernetes.io/projected/c490a42c-18b6-408a-baad-ce967460f21c-kube-api-access-2dbpr\") pod \"payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:55.770457 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.770423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c490a42c-18b6-408a-baad-ce967460f21c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:55.871336 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.871294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c490a42c-18b6-408a-baad-ce967460f21c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:55.871537 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.871357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbpr\" (UniqueName: \"kubernetes.io/projected/c490a42c-18b6-408a-baad-ce967460f21c-kube-api-access-2dbpr\") pod \"payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:55.872071 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.872050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c490a42c-18b6-408a-baad-ce967460f21c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:55.884301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.884270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbpr\" (UniqueName: \"kubernetes.io/projected/c490a42c-18b6-408a-baad-ce967460f21c-kube-api-access-2dbpr\") pod \"payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:55.944483 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:55.944400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:56.024016 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.023984 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:56.052805 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.052774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:56.239917 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.239826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:56.301190 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.301164 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z"] Sep 10 19:27:56.302844 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:27:56.302816 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc490a42c_18b6_408a_baad_ce967460f21c.slice/crio-bdef74c43cbf1f69594593ec412916c7249547fa5b32b4ec402ff0474cd3a7a0 WatchSource:0}: Error finding container bdef74c43cbf1f69594593ec412916c7249547fa5b32b4ec402ff0474cd3a7a0: Status 404 returned error can't find the container with id bdef74c43cbf1f69594593ec412916c7249547fa5b32b4ec402ff0474cd3a7a0 Sep 10 19:27:56.383650 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.383617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" event={"ID":"c490a42c-18b6-408a-baad-ce967460f21c","Type":"ContainerStarted","Data":"1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493"} Sep 10 19:27:56.383794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.383716 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:27:56.383794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.383743 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" event={"ID":"c490a42c-18b6-408a-baad-ce967460f21c","Type":"ContainerStarted","Data":"bdef74c43cbf1f69594593ec412916c7249547fa5b32b4ec402ff0474cd3a7a0"} Sep 10 19:27:56.405797 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.405751 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" podStartSLOduration=1.405735112 podStartE2EDuration="1.405735112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:56.403609836 +0000 UTC m=+1042.320265865" watchObservedRunningTime="2026-09-10 19:27:56.405735112 +0000 UTC m=+1042.322391141" Sep 10 19:27:56.581463 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.581419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:56.604526 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.604492 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" path="/var/lib/kubelet/pods/6f5b1544-0e93-4aba-b06c-900c311b3db8/volumes" Sep 10 19:27:56.764317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.764289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:27:56.957771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:27:56.957692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:00.207932 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:00.207899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:00.407534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:00.407500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:01.552397 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:01.551356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:01.846444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:01.846326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:02.294132 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:02.294101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:02.840190 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:02.840107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:03.001328 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.001299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:03.153655 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.153531 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd"] Sep 10 19:28:03.154180 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.154153 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" Sep 10 19:28:03.154286 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.154184 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5b1544-0e93-4aba-b06c-900c311b3db8" containerName="istio-proxy" Sep 10 19:28:03.157961 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.157938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.161849 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.161831 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-94ca92-dockercfg-mfls7\"" Sep 10 19:28:03.165647 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.165631 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-94ca92\"" Sep 10 19:28:03.186836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.186811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd"] Sep 10 19:28:03.247062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.246986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ef0e1c-2af6-4867-b132-e1a873445171-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.247355 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.247330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5dq\" (UniqueName: \"kubernetes.io/projected/e1ef0e1c-2af6-4867-b132-e1a873445171-kube-api-access-5p5dq\") pod \"payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.353262 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.351877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ef0e1c-2af6-4867-b132-e1a873445171-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.353262 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.351945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5dq\" (UniqueName: \"kubernetes.io/projected/e1ef0e1c-2af6-4867-b132-e1a873445171-kube-api-access-5p5dq\") pod \"payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.353262 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.353159 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ef0e1c-2af6-4867-b132-e1a873445171-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.368744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.368721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5dq\" (UniqueName: \"kubernetes.io/projected/e1ef0e1c-2af6-4867-b132-e1a873445171-kube-api-access-5p5dq\") pod \"payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.467862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.467777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:03.763419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:03.763390 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd"] Sep 10 19:28:03.765475 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:28:03.765399 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ef0e1c_2af6_4867_b132_e1a873445171.slice/crio-fe82d5cb57375762a7419aff241a560341b7f68744f30c46c68e64a0a6a637fa WatchSource:0}: Error finding container fe82d5cb57375762a7419aff241a560341b7f68744f30c46c68e64a0a6a637fa: Status 404 returned error can't find the container with id fe82d5cb57375762a7419aff241a560341b7f68744f30c46c68e64a0a6a637fa Sep 10 19:28:04.437694 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:04.437048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" event={"ID":"e1ef0e1c-2af6-4867-b132-e1a873445171","Type":"ContainerStarted","Data":"83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8"} Sep 10 19:28:04.437694 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:04.437110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" event={"ID":"e1ef0e1c-2af6-4867-b132-e1a873445171","Type":"ContainerStarted","Data":"fe82d5cb57375762a7419aff241a560341b7f68744f30c46c68e64a0a6a637fa"} Sep 10 19:28:04.437694 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:04.437343 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:04.466968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:04.466902 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" podStartSLOduration=1.466883424 podStartE2EDuration="1.466883424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:04.465357007 +0000 UTC m=+1050.382013033" watchObservedRunningTime="2026-09-10 19:28:04.466883424 +0000 UTC m=+1050.383539453" Sep 10 19:28:07.393292 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:07.393256 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:28:08.394565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:08.394516 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 19:28:08.395050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:08.394584 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 19:28:09.474800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:09.474765 2576 generic.go:356] "Generic (PLEG): container finished" podID="01845e69-4217-4e5e-b6b7-c255f3325067" containerID="55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5" exitCode=0 Sep 10 19:28:09.475209 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:09.474840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" event={"ID":"01845e69-4217-4e5e-b6b7-c255f3325067","Type":"ContainerDied","Data":"55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5"} Sep 10 19:28:09.475251 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:09.475217 2576 scope.go:117] "RemoveContainer" containerID="55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5" Sep 10 19:28:09.476315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:09.476288 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:28:09.476407 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:09.476356 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:28:09.476450 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:09.476414 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:28:10.482882 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:10.482845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" event={"ID":"01845e69-4217-4e5e-b6b7-c255f3325067","Type":"ContainerStarted","Data":"70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3"} Sep 10 19:28:11.398520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:11.398486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:28:11.398742 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:11.398720 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 10 19:28:11.398800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:11.398772 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 10 19:28:12.399392 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:12.399340 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 10 19:28:12.399868 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:12.399432 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 10 19:28:13.398753 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:13.398655 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 10 19:28:13.398753 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:13.398718 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 10 19:28:14.399479 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:14.399440 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 10 19:28:14.399902 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:14.399513 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 10 19:28:15.398902 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:15.398861 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 10 19:28:15.399120 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:15.398943 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 10 19:28:15.449775 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:15.449741 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:28:16.128038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:16.128007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:16.402498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:16.402416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:28:16.402706 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:16.402679 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:28:16.403417 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:16.403400 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:28:18.252962 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:18.252925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:18.568903 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:18.568871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:18.993072 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:18.992993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:19.209590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:19.209549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:19.712149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:19.712122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:22.208876 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:22.208835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:22.316384 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:22.316350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:23.352498 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:23.352469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:23.452226 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:23.452189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:23.627828 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:23.627745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:24.069340 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.069310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:24.183457 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.183423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:24.329802 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.326669 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj"] Sep 10 19:28:24.332584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.332557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.336021 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.336000 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f15c09\"" Sep 10 19:28:24.336201 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.336184 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f15c09-dockercfg-6whmw\"" Sep 10 19:28:24.355395 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.355341 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8"] Sep 10 19:28:24.359604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.359528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83610d2-37ff-45c1-9d84-3d2603461afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.359754 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.359725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9v2\" (UniqueName: \"kubernetes.io/projected/c83610d2-37ff-45c1-9d84-3d2603461afd-kube-api-access-rz9v2\") pod \"payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.361195 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.361171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj"] Sep 10 19:28:24.361844 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.361307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.369674 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.369639 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8"] Sep 10 19:28:24.460779 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.460743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9v2\" (UniqueName: \"kubernetes.io/projected/c83610d2-37ff-45c1-9d84-3d2603461afd-kube-api-access-rz9v2\") pod \"payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.460969 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.460798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83610d2-37ff-45c1-9d84-3d2603461afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.460969 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.460830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5356b9-f215-4857-a314-d4c0fb0edc69-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.460969 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.460879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nbl\" (UniqueName: \"kubernetes.io/projected/ad5356b9-f215-4857-a314-d4c0fb0edc69-kube-api-access-d8nbl\") pod \"payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.461610 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.461592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83610d2-37ff-45c1-9d84-3d2603461afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.471388 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.471353 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9v2\" (UniqueName: \"kubernetes.io/projected/c83610d2-37ff-45c1-9d84-3d2603461afd-kube-api-access-rz9v2\") pod \"payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.561794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.561757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5356b9-f215-4857-a314-d4c0fb0edc69-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.561794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.561802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nbl\" (UniqueName: \"kubernetes.io/projected/ad5356b9-f215-4857-a314-d4c0fb0edc69-kube-api-access-d8nbl\") pod \"payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.562556 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.562532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5356b9-f215-4857-a314-d4c0fb0edc69-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.570537 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.570510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nbl\" (UniqueName: \"kubernetes.io/projected/ad5356b9-f215-4857-a314-d4c0fb0edc69-kube-api-access-d8nbl\") pod \"payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.655946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.655853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:24.679803 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.679781 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:24.802291 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.802259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:24.933715 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.933693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj"] Sep 10 19:28:24.934508 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:28:24.934481 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83610d2_37ff_45c1_9d84_3d2603461afd.slice/crio-683be2ec8e37d6accd40d7298abd6ba10176253647e746417adcd07684f54651 WatchSource:0}: Error finding container 683be2ec8e37d6accd40d7298abd6ba10176253647e746417adcd07684f54651: Status 404 returned error can't find the container with id 683be2ec8e37d6accd40d7298abd6ba10176253647e746417adcd07684f54651 Sep 10 19:28:24.966479 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:24.966455 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8"] Sep 10 19:28:24.971579 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:28:24.971548 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5356b9_f215_4857_a314_d4c0fb0edc69.slice/crio-abf3b1e8475eb1117af5f79751c23af6b1f28f8b6a9f8dbe28186c575b9be82d WatchSource:0}: Error finding container abf3b1e8475eb1117af5f79751c23af6b1f28f8b6a9f8dbe28186c575b9be82d: Status 404 returned error can't find the container with id abf3b1e8475eb1117af5f79751c23af6b1f28f8b6a9f8dbe28186c575b9be82d Sep 10 19:28:25.581460 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.581422 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" event={"ID":"ad5356b9-f215-4857-a314-d4c0fb0edc69","Type":"ContainerStarted","Data":"22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde"} Sep 10 19:28:25.581910 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.581491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" event={"ID":"ad5356b9-f215-4857-a314-d4c0fb0edc69","Type":"ContainerStarted","Data":"abf3b1e8475eb1117af5f79751c23af6b1f28f8b6a9f8dbe28186c575b9be82d"} Sep 10 19:28:25.581910 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.581519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:25.583243 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.583212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" event={"ID":"c83610d2-37ff-45c1-9d84-3d2603461afd","Type":"ContainerStarted","Data":"9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4"} Sep 10 19:28:25.583396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.583263 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" event={"ID":"c83610d2-37ff-45c1-9d84-3d2603461afd","Type":"ContainerStarted","Data":"683be2ec8e37d6accd40d7298abd6ba10176253647e746417adcd07684f54651"} Sep 10 19:28:25.583396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.583340 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:25.604997 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.604945 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" podStartSLOduration=1.604924169 podStartE2EDuration="1.604924169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:25.602593211 +0000 UTC m=+1071.519249241" watchObservedRunningTime="2026-09-10 19:28:25.604924169 +0000 UTC m=+1071.521580197" Sep 10 19:28:25.626099 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.626048 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" podStartSLOduration=1.6260331300000002 podStartE2EDuration="1.62603313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:25.624427013 +0000 UTC m=+1071.541083045" watchObservedRunningTime="2026-09-10 19:28:25.62603313 +0000 UTC m=+1071.542689159" Sep 10 19:28:25.643770 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:25.643745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:26.461479 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:26.461440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:30.230201 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:30.230167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:35.476216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:35.476184 2576 scope.go:117] "RemoveContainer" containerID="85e3de1d82c8e83a0afa6067cb61195d8234a2625d9a6a4a717e84a988a79bc7" Sep 10 19:28:35.485176 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:35.485159 2576 scope.go:117] "RemoveContainer" containerID="d490621ab7b24eb076ebc24a0339a9712998e56eaba272af4b42381353268c04" Sep 10 19:28:36.591558 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:36.591526 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:28:36.591959 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:36.591588 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:28:41.654937 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:41.654904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:42.037640 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:42.037596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:42.310695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:42.310603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:42.974947 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:42.974917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:44.982906 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:44.982877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:45.448475 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:45.448439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:45.611520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:45.611482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:47.075879 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.075850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:47.176955 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.176917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:47.278840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.278810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:47.761651 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.761624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:47.867962 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.867932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:28:47.876599 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.876571 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7"] Sep 10 19:28:47.881734 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.881716 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:47.884496 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.884476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-84cb5814-dockercfg-k2d65\"" Sep 10 19:28:47.886525 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.886511 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-84cb5814\"" Sep 10 19:28:47.894560 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.894540 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5"] Sep 10 19:28:47.898014 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.897995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:47.901799 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.901778 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7"] Sep 10 19:28:47.921135 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.921111 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5"] Sep 10 19:28:47.979971 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.979938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e8ee5-5147-4177-810a-53ebbaf5f252-plugins-config-volume\") pod \"payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:47.980137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.980013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6mgq\" (UniqueName: \"kubernetes.io/projected/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-kube-api-access-l6mgq\") pod \"payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:47.980137 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.980086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:47.980218 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:47.980186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n96\" (UniqueName: \"kubernetes.io/projected/af9e8ee5-5147-4177-810a-53ebbaf5f252-kube-api-access-v2n96\") pod \"payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:48.081459 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.081423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6mgq\" (UniqueName: \"kubernetes.io/projected/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-kube-api-access-l6mgq\") pod \"payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:48.081933 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.081467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:48.081933 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.081508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n96\" (UniqueName: \"kubernetes.io/projected/af9e8ee5-5147-4177-810a-53ebbaf5f252-kube-api-access-v2n96\") pod \"payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:48.081933 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.081551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e8ee5-5147-4177-810a-53ebbaf5f252-plugins-config-volume\") pod \"payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:48.082219 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.082199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e8ee5-5147-4177-810a-53ebbaf5f252-plugins-config-volume\") pod \"payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:48.082260 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.082211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:48.091078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.091045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n96\" (UniqueName: \"kubernetes.io/projected/af9e8ee5-5147-4177-810a-53ebbaf5f252-kube-api-access-v2n96\") pod \"payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:48.091230 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.091181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6mgq\" (UniqueName: \"kubernetes.io/projected/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-kube-api-access-l6mgq\") pod \"payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:48.192497 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.192450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:48.208790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.208758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:48.475086 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.475060 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7"] Sep 10 19:28:48.476640 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:28:48.476594 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb8bc7b_ea05_4cc1_9071_eb5b396b9102.slice/crio-3cd688b5cfda675c75003366665990c8ef136055e5a7bf65efa62d6246d01bcb WatchSource:0}: Error finding container 3cd688b5cfda675c75003366665990c8ef136055e5a7bf65efa62d6246d01bcb: Status 404 returned error can't find the container with id 3cd688b5cfda675c75003366665990c8ef136055e5a7bf65efa62d6246d01bcb Sep 10 19:28:48.498681 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.498561 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5"] Sep 10 19:28:48.500566 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:28:48.500541 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9e8ee5_5147_4177_810a_53ebbaf5f252.slice/crio-0c5095f96c878c0f70e7a54a21b852835d7be8e39c2431ebd334c4d5dea291d1 WatchSource:0}: Error finding container 0c5095f96c878c0f70e7a54a21b852835d7be8e39c2431ebd334c4d5dea291d1: Status 404 returned error can't find the container with id 0c5095f96c878c0f70e7a54a21b852835d7be8e39c2431ebd334c4d5dea291d1 Sep 10 19:28:48.723516 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.723420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" event={"ID":"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102","Type":"ContainerStarted","Data":"82b4c6dc6fab78b72952e148b99c136ba1386c595ce4eed88a397bf57187f2c9"} Sep 10 19:28:48.723516 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.723496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" event={"ID":"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102","Type":"ContainerStarted","Data":"3cd688b5cfda675c75003366665990c8ef136055e5a7bf65efa62d6246d01bcb"} Sep 10 19:28:48.723730 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.723528 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:48.725200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.725175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" event={"ID":"af9e8ee5-5147-4177-810a-53ebbaf5f252","Type":"ContainerStarted","Data":"8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5"} Sep 10 19:28:48.725316 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.725216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" event={"ID":"af9e8ee5-5147-4177-810a-53ebbaf5f252","Type":"ContainerStarted","Data":"0c5095f96c878c0f70e7a54a21b852835d7be8e39c2431ebd334c4d5dea291d1"} Sep 10 19:28:48.725316 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.725278 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:28:48.756953 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.756901 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" podStartSLOduration=1.756885681 podStartE2EDuration="1.756885681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:48.755771412 +0000 UTC m=+1094.672427441" watchObservedRunningTime="2026-09-10 19:28:48.756885681 +0000 UTC m=+1094.673541705" Sep 10 19:28:48.796813 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:48.796770 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" podStartSLOduration=1.7967537660000001 podStartE2EDuration="1.796753766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:48.79552919 +0000 UTC m=+1094.712185220" watchObservedRunningTime="2026-09-10 19:28:48.796753766 +0000 UTC m=+1094.713409794" Sep 10 19:28:59.740499 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:59.740472 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:28:59.740894 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:28:59.740541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:29:01.688253 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:01.688213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:29:10.394065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:10.394023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:29:21.729952 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:21.729914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:29:26.522582 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:26.522549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:29:29.813009 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:29.812957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:29:35.521321 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:35.521286 2576 scope.go:117] "RemoveContainer" containerID="98a43f89ff9a432070553153fc7fa19f63e1cf478d69e967223e8091d2bcf029" Sep 10 19:29:35.530598 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:35.530581 2576 scope.go:117] "RemoveContainer" containerID="3cb322fefe667afaec751e4a4070fec294ac1e131f746d75e2dba1a0f0714de9" Sep 10 19:29:49.998140 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:49.998106 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4"] Sep 10 19:29:50.003536 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.003510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" Sep 10 19:29:50.006271 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.006242 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:29:50.006432 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.006351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:29:50.012976 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.012951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4"] Sep 10 19:29:50.142117 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.142081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c3dc2-bcf7-48d9-9894-fa1a2a391656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4\" (UID: \"314c3dc2-bcf7-48d9-9894-fa1a2a391656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" Sep 10 19:29:50.229519 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.229490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:29:50.243794 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.243764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c3dc2-bcf7-48d9-9894-fa1a2a391656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4\" (UID: \"314c3dc2-bcf7-48d9-9894-fa1a2a391656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" Sep 10 19:29:50.244528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.244507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c3dc2-bcf7-48d9-9894-fa1a2a391656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4\" (UID: \"314c3dc2-bcf7-48d9-9894-fa1a2a391656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" Sep 10 19:29:50.315315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.315287 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" Sep 10 19:29:50.589401 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:50.589344 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4"] Sep 10 19:29:50.590625 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:29:50.590596 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314c3dc2_bcf7_48d9_9894_fa1a2a391656.slice/crio-8cee95318baf59b7acbbc477855ceb3149a590ba57818469e17c0b8d515b5841 WatchSource:0}: Error finding container 8cee95318baf59b7acbbc477855ceb3149a590ba57818469e17c0b8d515b5841: Status 404 returned error can't find the container with id 8cee95318baf59b7acbbc477855ceb3149a590ba57818469e17c0b8d515b5841 Sep 10 19:29:51.107210 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:51.107121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" event={"ID":"314c3dc2-bcf7-48d9-9894-fa1a2a391656","Type":"ContainerStarted","Data":"1ce2f831dcc3c10eca820cc4956a31a016894d6b9f0f437a0cbbbfabfc446595"} Sep 10 19:29:51.107210 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:51.107184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" event={"ID":"314c3dc2-bcf7-48d9-9894-fa1a2a391656","Type":"ContainerStarted","Data":"8cee95318baf59b7acbbc477855ceb3149a590ba57818469e17c0b8d515b5841"} Sep 10 19:29:51.130062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:51.130002 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" podStartSLOduration=2.129988045 podStartE2EDuration="2.129988045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:51.127679812 +0000 UTC m=+1157.044335841" watchObservedRunningTime="2026-09-10 19:29:51.129988045 +0000 UTC m=+1157.046644073" Sep 10 19:29:52.115721 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:52.115681 2576 generic.go:356] "Generic (PLEG): container finished" podID="314c3dc2-bcf7-48d9-9894-fa1a2a391656" containerID="1ce2f831dcc3c10eca820cc4956a31a016894d6b9f0f437a0cbbbfabfc446595" exitCode=0 Sep 10 19:29:52.116150 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:52.115769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" event={"ID":"314c3dc2-bcf7-48d9-9894-fa1a2a391656","Type":"ContainerDied","Data":"1ce2f831dcc3c10eca820cc4956a31a016894d6b9f0f437a0cbbbfabfc446595"} Sep 10 19:29:53.253633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:53.253608 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" Sep 10 19:29:53.265710 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:53.265687 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c3dc2-bcf7-48d9-9894-fa1a2a391656-maas-api-service-ca\") pod \"314c3dc2-bcf7-48d9-9894-fa1a2a391656\" (UID: \"314c3dc2-bcf7-48d9-9894-fa1a2a391656\") " Sep 10 19:29:53.266025 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:53.266007 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314c3dc2-bcf7-48d9-9894-fa1a2a391656-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "314c3dc2-bcf7-48d9-9894-fa1a2a391656" (UID: "314c3dc2-bcf7-48d9-9894-fa1a2a391656"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:53.366776 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:53.366740 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c3dc2-bcf7-48d9-9894-fa1a2a391656-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:29:54.129119 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:54.129091 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" Sep 10 19:29:54.129297 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:54.129134 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4" event={"ID":"314c3dc2-bcf7-48d9-9894-fa1a2a391656","Type":"ContainerDied","Data":"8cee95318baf59b7acbbc477855ceb3149a590ba57818469e17c0b8d515b5841"} Sep 10 19:29:54.129297 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:29:54.129179 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cee95318baf59b7acbbc477855ceb3149a590ba57818469e17c0b8d515b5841" Sep 10 19:30:00.203959 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.203880 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9"] Sep 10 19:30:00.204327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.204313 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314c3dc2-bcf7-48d9-9894-fa1a2a391656" containerName="revoke-keys" Sep 10 19:30:00.204398 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.204331 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="314c3dc2-bcf7-48d9-9894-fa1a2a391656" containerName="revoke-keys" Sep 10 19:30:00.204447 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.204437 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="314c3dc2-bcf7-48d9-9894-fa1a2a391656" containerName="revoke-keys" Sep 10 19:30:00.207578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.207547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" Sep 10 19:30:00.212145 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.212117 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:30:00.213760 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.213736 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg"] Sep 10 19:30:00.218189 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.218159 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" Sep 10 19:30:00.223117 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.222899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" Sep 10 19:30:00.226444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.225786 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq"] Sep 10 19:30:00.245580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.245552 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd"] Sep 10 19:30:00.245764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.245736 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" Sep 10 19:30:00.251698 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.251438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9"] Sep 10 19:30:00.251698 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.251481 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd"] Sep 10 19:30:00.251698 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.251503 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq"] Sep 10 19:30:00.251698 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.251589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" Sep 10 19:30:00.282340 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.282130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" Sep 10 19:30:00.284900 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.283237 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg"] Sep 10 19:30:00.295443 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.295413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" Sep 10 19:30:00.348609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.348365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" Sep 10 19:30:00.555727 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.555700 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9"] Sep 10 19:30:00.557215 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:30:00.557183 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0070b21_bb9c_42b2_90c6_3001087a798a.slice/crio-238eb5dad29fd2b9f44a8abf4b15fe9570074a19076f03f47ce66d3d87a6f554 WatchSource:0}: Error finding container 238eb5dad29fd2b9f44a8abf4b15fe9570074a19076f03f47ce66d3d87a6f554: Status 404 returned error can't find the container with id 238eb5dad29fd2b9f44a8abf4b15fe9570074a19076f03f47ce66d3d87a6f554 Sep 10 19:30:00.639156 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.639131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg"] Sep 10 19:30:00.640668 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:30:00.640642 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dfe84f_62b4_4569_b673_22ed1419162d.slice/crio-666b9bd3ae189f613eb62963666f3b7b2e333cef83c35233f0a8725db360a447 WatchSource:0}: Error finding container 666b9bd3ae189f613eb62963666f3b7b2e333cef83c35233f0a8725db360a447: Status 404 returned error can't find the container with id 666b9bd3ae189f613eb62963666f3b7b2e333cef83c35233f0a8725db360a447 Sep 10 19:30:00.642177 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.642152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq"] Sep 10 19:30:00.642874 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:30:00.642857 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8f0063_5d60_4afd_86d5_489c600a623b.slice/crio-20e46882cb50c553860363af532187c2b980e4ac6553dbc0436b9c0a1e31b67e WatchSource:0}: Error finding container 20e46882cb50c553860363af532187c2b980e4ac6553dbc0436b9c0a1e31b67e: Status 404 returned error can't find the container with id 20e46882cb50c553860363af532187c2b980e4ac6553dbc0436b9c0a1e31b67e Sep 10 19:30:00.671733 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:00.671709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd"] Sep 10 19:30:00.673437 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:30:00.673414 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55935be2_3e6b_40d5_98fc_bb322a5fb2f9.slice/crio-5058fe7356dc0c983dcdc2d0145cba3bf3eee75b9a9e7d4ec90b20c569ef97f7 WatchSource:0}: Error finding container 5058fe7356dc0c983dcdc2d0145cba3bf3eee75b9a9e7d4ec90b20c569ef97f7: Status 404 returned error can't find the container with id 5058fe7356dc0c983dcdc2d0145cba3bf3eee75b9a9e7d4ec90b20c569ef97f7 Sep 10 19:30:01.170574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.170537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" event={"ID":"1e8f0063-5d60-4afd-86d5-489c600a623b","Type":"ContainerStarted","Data":"bafe2a9b29a8b910f6ffc39301b4b7a5838833d8601870f2d42cfa800be19e68"} Sep 10 19:30:01.170761 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.170597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" event={"ID":"1e8f0063-5d60-4afd-86d5-489c600a623b","Type":"ContainerStarted","Data":"20e46882cb50c553860363af532187c2b980e4ac6553dbc0436b9c0a1e31b67e"} Sep 10 19:30:01.172075 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.172049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" event={"ID":"55935be2-3e6b-40d5-98fc-bb322a5fb2f9","Type":"ContainerStarted","Data":"060c1361d190ab9416c9f6c3e415945393a57987624103ea1630a93087c838db"} Sep 10 19:30:01.172197 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.172098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" event={"ID":"55935be2-3e6b-40d5-98fc-bb322a5fb2f9","Type":"ContainerStarted","Data":"5058fe7356dc0c983dcdc2d0145cba3bf3eee75b9a9e7d4ec90b20c569ef97f7"} Sep 10 19:30:01.173519 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.173496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" event={"ID":"93dfe84f-62b4-4569-b673-22ed1419162d","Type":"ContainerStarted","Data":"37906a48607690418dabddd68aaa0ba422e5b4256c44a005a4411bcfd4d2940f"} Sep 10 19:30:01.173630 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.173534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" event={"ID":"93dfe84f-62b4-4569-b673-22ed1419162d","Type":"ContainerStarted","Data":"666b9bd3ae189f613eb62963666f3b7b2e333cef83c35233f0a8725db360a447"} Sep 10 19:30:01.175116 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.175095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" event={"ID":"a0070b21-bb9c-42b2-90c6-3001087a798a","Type":"ContainerStarted","Data":"9780b1de24d8ae02510939bc0fb1cf05ea0c16014e4599e937d1a4b085a2b51d"} Sep 10 19:30:01.175212 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.175135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" event={"ID":"a0070b21-bb9c-42b2-90c6-3001087a798a","Type":"ContainerStarted","Data":"238eb5dad29fd2b9f44a8abf4b15fe9570074a19076f03f47ce66d3d87a6f554"} Sep 10 19:30:01.187940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.187896 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" podStartSLOduration=1.18788166 podStartE2EDuration="1.18788166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:01.186209851 +0000 UTC m=+1167.102865888" watchObservedRunningTime="2026-09-10 19:30:01.18788166 +0000 UTC m=+1167.104537689" Sep 10 19:30:01.203411 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.203340 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" podStartSLOduration=1.203328171 podStartE2EDuration="1.203328171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:01.201521504 +0000 UTC m=+1167.118177534" watchObservedRunningTime="2026-09-10 19:30:01.203328171 +0000 UTC m=+1167.119984241" Sep 10 19:30:01.217853 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.217805 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" podStartSLOduration=1.2177916579999999 podStartE2EDuration="1.217791658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:01.215779471 +0000 UTC m=+1167.132435502" watchObservedRunningTime="2026-09-10 19:30:01.217791658 +0000 UTC m=+1167.134447686" Sep 10 19:30:01.234122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:01.234052 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" podStartSLOduration=1.2340317729999999 podStartE2EDuration="1.234031773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:01.230890272 +0000 UTC m=+1167.147546301" watchObservedRunningTime="2026-09-10 19:30:01.234031773 +0000 UTC m=+1167.150687805" Sep 10 19:30:02.183167 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.183132 2576 generic.go:356] "Generic (PLEG): container finished" podID="93dfe84f-62b4-4569-b673-22ed1419162d" containerID="37906a48607690418dabddd68aaa0ba422e5b4256c44a005a4411bcfd4d2940f" exitCode=0 Sep 10 19:30:02.183353 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.183208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" event={"ID":"93dfe84f-62b4-4569-b673-22ed1419162d","Type":"ContainerDied","Data":"37906a48607690418dabddd68aaa0ba422e5b4256c44a005a4411bcfd4d2940f"} Sep 10 19:30:02.184887 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.184862 2576 generic.go:356] "Generic (PLEG): container finished" podID="a0070b21-bb9c-42b2-90c6-3001087a798a" containerID="9780b1de24d8ae02510939bc0fb1cf05ea0c16014e4599e937d1a4b085a2b51d" exitCode=0 Sep 10 19:30:02.184972 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.184945 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" event={"ID":"a0070b21-bb9c-42b2-90c6-3001087a798a","Type":"ContainerDied","Data":"9780b1de24d8ae02510939bc0fb1cf05ea0c16014e4599e937d1a4b085a2b51d"} Sep 10 19:30:02.186502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.186482 2576 generic.go:356] "Generic (PLEG): container finished" podID="1e8f0063-5d60-4afd-86d5-489c600a623b" containerID="bafe2a9b29a8b910f6ffc39301b4b7a5838833d8601870f2d42cfa800be19e68" exitCode=0 Sep 10 19:30:02.186606 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.186543 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" event={"ID":"1e8f0063-5d60-4afd-86d5-489c600a623b","Type":"ContainerDied","Data":"bafe2a9b29a8b910f6ffc39301b4b7a5838833d8601870f2d42cfa800be19e68"} Sep 10 19:30:02.187932 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.187913 2576 generic.go:356] "Generic (PLEG): container finished" podID="55935be2-3e6b-40d5-98fc-bb322a5fb2f9" containerID="060c1361d190ab9416c9f6c3e415945393a57987624103ea1630a93087c838db" exitCode=0 Sep 10 19:30:02.188029 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:02.188004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" event={"ID":"55935be2-3e6b-40d5-98fc-bb322a5fb2f9","Type":"ContainerDied","Data":"060c1361d190ab9416c9f6c3e415945393a57987624103ea1630a93087c838db"} Sep 10 19:30:03.386604 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:03.386577 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" Sep 10 19:30:03.388567 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:03.388543 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" Sep 10 19:30:03.444710 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:03.444678 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" Sep 10 19:30:03.448236 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:03.448216 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" Sep 10 19:30:04.203603 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.203573 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" Sep 10 19:30:04.203786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.203570 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg" event={"ID":"93dfe84f-62b4-4569-b673-22ed1419162d","Type":"ContainerDied","Data":"666b9bd3ae189f613eb62963666f3b7b2e333cef83c35233f0a8725db360a447"} Sep 10 19:30:04.203786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.203686 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666b9bd3ae189f613eb62963666f3b7b2e333cef83c35233f0a8725db360a447" Sep 10 19:30:04.205028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.205004 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" Sep 10 19:30:04.205028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.205009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-6g2b9" event={"ID":"a0070b21-bb9c-42b2-90c6-3001087a798a","Type":"ContainerDied","Data":"238eb5dad29fd2b9f44a8abf4b15fe9570074a19076f03f47ce66d3d87a6f554"} Sep 10 19:30:04.205196 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.205053 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="238eb5dad29fd2b9f44a8abf4b15fe9570074a19076f03f47ce66d3d87a6f554" Sep 10 19:30:04.206565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.206548 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" Sep 10 19:30:04.206684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.206538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq" event={"ID":"1e8f0063-5d60-4afd-86d5-489c600a623b","Type":"ContainerDied","Data":"20e46882cb50c553860363af532187c2b980e4ac6553dbc0436b9c0a1e31b67e"} Sep 10 19:30:04.206684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.206670 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e46882cb50c553860363af532187c2b980e4ac6553dbc0436b9c0a1e31b67e" Sep 10 19:30:04.208048 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.208027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" event={"ID":"55935be2-3e6b-40d5-98fc-bb322a5fb2f9","Type":"ContainerDied","Data":"5058fe7356dc0c983dcdc2d0145cba3bf3eee75b9a9e7d4ec90b20c569ef97f7"} Sep 10 19:30:04.208162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.208068 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5058fe7356dc0c983dcdc2d0145cba3bf3eee75b9a9e7d4ec90b20c569ef97f7" Sep 10 19:30:04.208162 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:04.208035 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd" Sep 10 19:30:05.882558 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:05.882526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:30:05.908289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:05.908262 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5"] Sep 10 19:30:05.908511 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:05.908486 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" podUID="af9e8ee5-5147-4177-810a-53ebbaf5f252" containerName="payload-processing" containerID="cri-o://8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5" gracePeriod=30 Sep 10 19:30:05.918928 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:05.918392 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7"] Sep 10 19:30:05.918928 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:05.918600 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" podUID="aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" containerName="payload-pre-processing" containerID="cri-o://82b4c6dc6fab78b72952e148b99c136ba1386c595ce4eed88a397bf57187f2c9" gracePeriod=30 Sep 10 19:30:06.128177 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.128141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:30:06.165592 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.165021 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4"] Sep 10 19:30:06.167706 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.167673 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84cb5814-9m2m4"] Sep 10 19:30:06.182614 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.182590 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:30:06.214834 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.214752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:30:06.225423 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.225393 2576 generic.go:356] "Generic (PLEG): container finished" podID="aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" containerID="82b4c6dc6fab78b72952e148b99c136ba1386c595ce4eed88a397bf57187f2c9" exitCode=0 Sep 10 19:30:06.225574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.225414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" event={"ID":"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102","Type":"ContainerDied","Data":"82b4c6dc6fab78b72952e148b99c136ba1386c595ce4eed88a397bf57187f2c9"} Sep 10 19:30:06.225574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.225511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" event={"ID":"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102","Type":"ContainerDied","Data":"3cd688b5cfda675c75003366665990c8ef136055e5a7bf65efa62d6246d01bcb"} Sep 10 19:30:06.225574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.225535 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd688b5cfda675c75003366665990c8ef136055e5a7bf65efa62d6246d01bcb" Sep 10 19:30:06.226211 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.226191 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:30:06.227405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.227111 2576 generic.go:356] "Generic (PLEG): container finished" podID="af9e8ee5-5147-4177-810a-53ebbaf5f252" containerID="8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5" exitCode=0 Sep 10 19:30:06.227405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.227287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" event={"ID":"af9e8ee5-5147-4177-810a-53ebbaf5f252","Type":"ContainerDied","Data":"8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5"} Sep 10 19:30:06.227405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.227328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" event={"ID":"af9e8ee5-5147-4177-810a-53ebbaf5f252","Type":"ContainerDied","Data":"0c5095f96c878c0f70e7a54a21b852835d7be8e39c2431ebd334c4d5dea291d1"} Sep 10 19:30:06.227405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.227357 2576 scope.go:117] "RemoveContainer" containerID="8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5" Sep 10 19:30:06.228345 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.227554 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5" Sep 10 19:30:06.242050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.242024 2576 scope.go:117] "RemoveContainer" containerID="8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5" Sep 10 19:30:06.242861 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:30:06.242837 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5\": container with ID starting with 8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5 not found: ID does not exist" containerID="8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5" Sep 10 19:30:06.242943 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.242874 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5"} err="failed to get container status \"8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5\": rpc error: code = NotFound desc = could not find container \"8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5\": container with ID starting with 8b36e434bec523e3f672e5bd932a52fdf5e4e9abb8aa55543fbd1b461e946cd5 not found: ID does not exist" Sep 10 19:30:06.284112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.284086 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-plugins-config-volume\") pod \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " Sep 10 19:30:06.284289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.284119 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e8ee5-5147-4177-810a-53ebbaf5f252-plugins-config-volume\") pod \"af9e8ee5-5147-4177-810a-53ebbaf5f252\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " Sep 10 19:30:06.284289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.284207 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2n96\" (UniqueName: \"kubernetes.io/projected/af9e8ee5-5147-4177-810a-53ebbaf5f252-kube-api-access-v2n96\") pod \"af9e8ee5-5147-4177-810a-53ebbaf5f252\" (UID: \"af9e8ee5-5147-4177-810a-53ebbaf5f252\") " Sep 10 19:30:06.284289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.284254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6mgq\" (UniqueName: \"kubernetes.io/projected/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-kube-api-access-l6mgq\") pod \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\" (UID: \"aeb8bc7b-ea05-4cc1-9071-eb5b396b9102\") " Sep 10 19:30:06.284501 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.284456 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" (UID: "aeb8bc7b-ea05-4cc1-9071-eb5b396b9102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:30:06.284558 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.284543 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:30:06.284596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.284568 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9e8ee5-5147-4177-810a-53ebbaf5f252-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af9e8ee5-5147-4177-810a-53ebbaf5f252" (UID: "af9e8ee5-5147-4177-810a-53ebbaf5f252"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:30:06.286510 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.286485 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-kube-api-access-l6mgq" (OuterVolumeSpecName: "kube-api-access-l6mgq") pod "aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" (UID: "aeb8bc7b-ea05-4cc1-9071-eb5b396b9102"). InnerVolumeSpecName "kube-api-access-l6mgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:30:06.286591 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.286511 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e8ee5-5147-4177-810a-53ebbaf5f252-kube-api-access-v2n96" (OuterVolumeSpecName: "kube-api-access-v2n96") pod "af9e8ee5-5147-4177-810a-53ebbaf5f252" (UID: "af9e8ee5-5147-4177-810a-53ebbaf5f252"). InnerVolumeSpecName "kube-api-access-v2n96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:30:06.385585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.385554 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9e8ee5-5147-4177-810a-53ebbaf5f252-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:30:06.385585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.385581 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2n96\" (UniqueName: \"kubernetes.io/projected/af9e8ee5-5147-4177-810a-53ebbaf5f252-kube-api-access-v2n96\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:30:06.385585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.385591 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6mgq\" (UniqueName: \"kubernetes.io/projected/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102-kube-api-access-l6mgq\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:30:06.555446 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.555420 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5"] Sep 10 19:30:06.564287 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.564255 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84cb5814-fdf74db7c-6wgs5"] Sep 10 19:30:06.603265 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.603229 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="314c3dc2-bcf7-48d9-9894-fa1a2a391656" path="/var/lib/kubelet/pods/314c3dc2-bcf7-48d9-9894-fa1a2a391656/volumes" Sep 10 19:30:06.603632 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:06.603614 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9e8ee5-5147-4177-810a-53ebbaf5f252" path="/var/lib/kubelet/pods/af9e8ee5-5147-4177-810a-53ebbaf5f252/volumes" Sep 10 19:30:07.234755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:07.234667 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7" Sep 10 19:30:07.253055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:07.253025 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7"] Sep 10 19:30:07.257920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:07.257895 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84cb5814-6d88b6ff86-pdxh7"] Sep 10 19:30:08.602324 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:08.602291 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" path="/var/lib/kubelet/pods/aeb8bc7b-ea05-4cc1-9071-eb5b396b9102/volumes" Sep 10 19:30:09.222746 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:09.222712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:30:14.298638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:14.298606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:30:34.630522 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:34.630492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:30:34.633764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:34.633744 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:30:35.556467 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:35.556437 2576 scope.go:117] "RemoveContainer" containerID="c1ce208791d2d56b48fd0e3fd4224c50f3ac7e542ff54ae5971f9b8649194142" Sep 10 19:30:35.565518 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:35.565495 2576 scope.go:117] "RemoveContainer" containerID="106d34c61662a726742aaa2868d84829f513a2dcc85ad5fb9bcb97bb984d783f" Sep 10 19:30:35.573612 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:35.573594 2576 scope.go:117] "RemoveContainer" containerID="81e98c9e75b3ca62c5e17ae34349599ff792f5937f842edc6e38218cd5739913" Sep 10 19:30:50.637221 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:50.637191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:30:51.311164 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:51.311127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:30:57.114114 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:30:57.114081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:20.715420 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:20.715362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:20.819887 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:20.819844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:20.971692 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:20.971616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:22.226973 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:22.226919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:22.318165 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:22.318134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:22.420441 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:22.420408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:23.017207 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:23.017141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:23.115684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:23.115649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:24.448415 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:24.448358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:29.672147 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:29.672062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:29.768014 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:29.767949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:31.147809 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:31.147774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:31.242085 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:31.242057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:31.812977 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:31.812944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:37.727644 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:37.727605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:37.873788 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:37.873755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:37.970749 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:37.970711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:48.453545 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:48.453502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:48.620106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:48.620076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:49.517249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:49.517215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:49.894783 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:49.894754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:31:50.441913 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:31:50.441882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:02.662537 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:02.662506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:09.534175 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:09.534132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:15.411751 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:15.411709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:15.486472 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:15.486434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:15.757722 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:15.757631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:16.159738 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:16.159704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:16.399225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:16.399192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:19.542568 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:19.542533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:19.641688 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:19.641655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:20.731779 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:20.731750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:20.802855 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:20.802822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:20.941918 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:20.941879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:21.228562 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.228524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:21.325366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.325330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:21.352736 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.352702 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p"] Sep 10 19:32:21.353125 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353113 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9e8ee5-5147-4177-810a-53ebbaf5f252" containerName="payload-processing" Sep 10 19:32:21.353170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353127 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9e8ee5-5147-4177-810a-53ebbaf5f252" containerName="payload-processing" Sep 10 19:32:21.353170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353142 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0070b21-bb9c-42b2-90c6-3001087a798a" containerName="cleanup" Sep 10 19:32:21.353170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353147 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0070b21-bb9c-42b2-90c6-3001087a798a" containerName="cleanup" Sep 10 19:32:21.353170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353153 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93dfe84f-62b4-4569-b673-22ed1419162d" containerName="cleanup" Sep 10 19:32:21.353170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353159 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="93dfe84f-62b4-4569-b673-22ed1419162d" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353172 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" containerName="payload-pre-processing" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353178 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" containerName="payload-pre-processing" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353184 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8f0063-5d60-4afd-86d5-489c600a623b" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353189 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8f0063-5d60-4afd-86d5-489c600a623b" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353200 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55935be2-3e6b-40d5-98fc-bb322a5fb2f9" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353204 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55935be2-3e6b-40d5-98fc-bb322a5fb2f9" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353268 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9e8ee5-5147-4177-810a-53ebbaf5f252" containerName="payload-processing" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353274 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="93dfe84f-62b4-4569-b673-22ed1419162d" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353292 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8f0063-5d60-4afd-86d5-489c600a623b" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353300 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0070b21-bb9c-42b2-90c6-3001087a798a" containerName="cleanup" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353307 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb8bc7b-ea05-4cc1-9071-eb5b396b9102" containerName="payload-pre-processing" Sep 10 19:32:21.353325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.353312 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55935be2-3e6b-40d5-98fc-bb322a5fb2f9" containerName="cleanup" Sep 10 19:32:21.356463 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.356444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.358896 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.358879 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd7dca\"" Sep 10 19:32:21.359003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.358896 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd7dca-dockercfg-78768\"" Sep 10 19:32:21.368849 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.368822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p"] Sep 10 19:32:21.419613 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.419584 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72"] Sep 10 19:32:21.420461 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.420429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c49716-dec6-4a3d-89b0-dcd4afc23344-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.420621 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.420562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5776v\" (UniqueName: \"kubernetes.io/projected/f3c49716-dec6-4a3d-89b0-dcd4afc23344-kube-api-access-5776v\") pod \"payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.423365 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.423345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.436732 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.436705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72"] Sep 10 19:32:21.521826 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.521736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5776v\" (UniqueName: \"kubernetes.io/projected/f3c49716-dec6-4a3d-89b0-dcd4afc23344-kube-api-access-5776v\") pod \"payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.521826 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.521781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9da5046-8fbe-439b-b221-cabeed0fb37c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.522069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.521827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c49716-dec6-4a3d-89b0-dcd4afc23344-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.522069 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.521929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9fm\" (UniqueName: \"kubernetes.io/projected/b9da5046-8fbe-439b-b221-cabeed0fb37c-kube-api-access-8n9fm\") pod \"payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.522629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.522605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c49716-dec6-4a3d-89b0-dcd4afc23344-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.531235 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.531210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5776v\" (UniqueName: \"kubernetes.io/projected/f3c49716-dec6-4a3d-89b0-dcd4afc23344-kube-api-access-5776v\") pod \"payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.623216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.623173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9fm\" (UniqueName: \"kubernetes.io/projected/b9da5046-8fbe-439b-b221-cabeed0fb37c-kube-api-access-8n9fm\") pod \"payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.623477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.623310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9da5046-8fbe-439b-b221-cabeed0fb37c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.624076 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.624050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9da5046-8fbe-439b-b221-cabeed0fb37c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.632366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.632327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9fm\" (UniqueName: \"kubernetes.io/projected/b9da5046-8fbe-439b-b221-cabeed0fb37c-kube-api-access-8n9fm\") pod \"payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.668147 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.668103 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:21.733899 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.733839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:21.954548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:21.954518 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p"] Sep 10 19:32:21.955427 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:32:21.955401 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c49716_dec6_4a3d_89b0_dcd4afc23344.slice/crio-e016f0e7a2adf5755941aeec455f72037ba95b611992c6f66f175e533ae9fdbf WatchSource:0}: Error finding container e016f0e7a2adf5755941aeec455f72037ba95b611992c6f66f175e533ae9fdbf: Status 404 returned error can't find the container with id e016f0e7a2adf5755941aeec455f72037ba95b611992c6f66f175e533ae9fdbf Sep 10 19:32:22.046859 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:22.046820 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72"] Sep 10 19:32:22.047870 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:32:22.047831 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9da5046_8fbe_439b_b221_cabeed0fb37c.slice/crio-f11af8c83ce026a107ca44d5c779619c87ccc337255699b861b8c54f77d9de9e WatchSource:0}: Error finding container f11af8c83ce026a107ca44d5c779619c87ccc337255699b861b8c54f77d9de9e: Status 404 returned error can't find the container with id f11af8c83ce026a107ca44d5c779619c87ccc337255699b861b8c54f77d9de9e Sep 10 19:32:22.049279 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:22.049252 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" event={"ID":"f3c49716-dec6-4a3d-89b0-dcd4afc23344","Type":"ContainerStarted","Data":"f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a"} Sep 10 19:32:22.049427 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:22.049303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" event={"ID":"f3c49716-dec6-4a3d-89b0-dcd4afc23344","Type":"ContainerStarted","Data":"e016f0e7a2adf5755941aeec455f72037ba95b611992c6f66f175e533ae9fdbf"} Sep 10 19:32:22.049427 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:22.049328 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:22.081698 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:22.081643 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" podStartSLOduration=1.081626012 podStartE2EDuration="1.081626012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:22.078673689 +0000 UTC m=+1307.995329718" watchObservedRunningTime="2026-09-10 19:32:22.081626012 +0000 UTC m=+1307.998282041" Sep 10 19:32:23.057689 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:23.057650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" event={"ID":"b9da5046-8fbe-439b-b221-cabeed0fb37c","Type":"ContainerStarted","Data":"001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d"} Sep 10 19:32:23.058204 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:23.057717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" event={"ID":"b9da5046-8fbe-439b-b221-cabeed0fb37c","Type":"ContainerStarted","Data":"f11af8c83ce026a107ca44d5c779619c87ccc337255699b861b8c54f77d9de9e"} Sep 10 19:32:23.058204 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:23.057883 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:23.076868 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:23.076813 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" podStartSLOduration=2.076789956 podStartE2EDuration="2.076789956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:23.074957755 +0000 UTC m=+1308.991613783" watchObservedRunningTime="2026-09-10 19:32:23.076789956 +0000 UTC m=+1308.993445986" Sep 10 19:32:33.059583 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:33.059550 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:32:34.076848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:34.076814 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:32:35.616366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:35.616325 2576 scope.go:117] "RemoveContainer" containerID="95a68adbb20e97a08207b70506d146e886f763f500d23cbe6097151b55c1ec93" Sep 10 19:32:39.776899 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:39.776867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:39.867816 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:39.867782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:44.466628 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:44.466592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:44.921104 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:44.921071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:45.017364 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.017331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:45.262843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.262759 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr"] Sep 10 19:32:45.268010 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.267978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.272645 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.272609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-66e9b4-openshift-default-dockercfg-qz5qn\"" Sep 10 19:32:45.300847 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.300811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr"] Sep 10 19:32:45.330185 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.330138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:45.341651 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-envoy\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.341756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd40f507-4b0a-410f-a820-ef1bd00566ee-istiod-ca-cert\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.341756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftjbs\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-kube-api-access-ftjbs\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.341848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-credential-socket\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.341848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-certs\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.341914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-podinfo\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.341914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-socket\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.341983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-data\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.342018 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.341981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-token\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443237 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-envoy\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd40f507-4b0a-410f-a820-ef1bd00566ee-istiod-ca-cert\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftjbs\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-kube-api-access-ftjbs\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-credential-socket\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-certs\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-podinfo\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-socket\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-data\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443756 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-token\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443911 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443777 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-credential-socket\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443911 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-certs\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.443979 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-socket\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.444017 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.443974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-data\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.444525 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.444503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd40f507-4b0a-410f-a820-ef1bd00566ee-istiod-ca-cert\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.445774 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.445751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-envoy\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.446102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.446082 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-podinfo\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.451962 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.451927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-token\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.452084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.452032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftjbs\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-kube-api-access-ftjbs\") pod \"e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.579197 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.579155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:45.867575 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.867547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr"] Sep 10 19:32:45.870879 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:32:45.870832 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd40f507_4b0a_410f_a820_ef1bd00566ee.slice/crio-a3f3d84fe86f43c4b163ab2ea09deb71c00ff5044e8f2588f5f5f049d634f605 WatchSource:0}: Error finding container a3f3d84fe86f43c4b163ab2ea09deb71c00ff5044e8f2588f5f5f049d634f605: Status 404 returned error can't find the container with id a3f3d84fe86f43c4b163ab2ea09deb71c00ff5044e8f2588f5f5f049d634f605 Sep 10 19:32:45.873934 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.873890 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:32:45.874014 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.873987 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:32:45.874052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.874030 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:32:45.902507 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:45.902479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:46.211935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:46.211831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" event={"ID":"fd40f507-4b0a-410f-a820-ef1bd00566ee","Type":"ContainerStarted","Data":"c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b"} Sep 10 19:32:46.211935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:46.211910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" event={"ID":"fd40f507-4b0a-410f-a820-ef1bd00566ee","Type":"ContainerStarted","Data":"a3f3d84fe86f43c4b163ab2ea09deb71c00ff5044e8f2588f5f5f049d634f605"} Sep 10 19:32:46.234881 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:46.234824 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" podStartSLOduration=1.234808458 podStartE2EDuration="1.234808458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:46.233113372 +0000 UTC m=+1332.149769400" watchObservedRunningTime="2026-09-10 19:32:46.234808458 +0000 UTC m=+1332.151464487" Sep 10 19:32:46.580052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:46.580013 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:46.585672 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:46.585646 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:47.217836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:47.217798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:47.218916 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:47.218894 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:32:48.154801 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:48.154766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:48.246791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:48.246753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:49.345687 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.345650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:49.446849 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.446814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:49.836811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.836773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:49.926577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.926543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:49.975057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.975010 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt"] Sep 10 19:32:49.979185 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.979157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:49.984127 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.984104 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-66e9b4\"" Sep 10 19:32:49.984275 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.984202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-66e9b4-dockercfg-fdq7d\"" Sep 10 19:32:49.993405 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:49.993361 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt"] Sep 10 19:32:50.026823 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.026787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:32:50.089810 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.089705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqlf\" (UniqueName: \"kubernetes.io/projected/4e951860-59a2-4953-adb4-df8ef7f8d959-kube-api-access-fpqlf\") pod \"payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:50.090011 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.089852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e951860-59a2-4953-adb4-df8ef7f8d959-plugins-config-volume\") pod \"payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:50.190861 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.190822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e951860-59a2-4953-adb4-df8ef7f8d959-plugins-config-volume\") pod \"payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:50.191050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.190914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqlf\" (UniqueName: \"kubernetes.io/projected/4e951860-59a2-4953-adb4-df8ef7f8d959-kube-api-access-fpqlf\") pod \"payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:50.191585 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.191565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e951860-59a2-4953-adb4-df8ef7f8d959-plugins-config-volume\") pod \"payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:50.202571 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.202545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqlf\" (UniqueName: \"kubernetes.io/projected/4e951860-59a2-4953-adb4-df8ef7f8d959-kube-api-access-fpqlf\") pod \"payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:50.290554 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.290506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:50.579488 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:50.579454 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt"] Sep 10 19:32:50.580463 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:32:50.580434 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e951860_59a2_4953_adb4_df8ef7f8d959.slice/crio-1f4ce787c27ad4f8321d177d6e9c97ff6ed521a306da91460fed9f0037f2bf9c WatchSource:0}: Error finding container 1f4ce787c27ad4f8321d177d6e9c97ff6ed521a306da91460fed9f0037f2bf9c: Status 404 returned error can't find the container with id 1f4ce787c27ad4f8321d177d6e9c97ff6ed521a306da91460fed9f0037f2bf9c Sep 10 19:32:51.244859 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:51.244818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" event={"ID":"4e951860-59a2-4953-adb4-df8ef7f8d959","Type":"ContainerStarted","Data":"4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37"} Sep 10 19:32:51.245090 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:51.244889 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:32:51.245090 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:51.244909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" event={"ID":"4e951860-59a2-4953-adb4-df8ef7f8d959","Type":"ContainerStarted","Data":"1f4ce787c27ad4f8321d177d6e9c97ff6ed521a306da91460fed9f0037f2bf9c"} Sep 10 19:32:51.264972 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:51.264905 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" podStartSLOduration=2.2648864570000002 podStartE2EDuration="2.264886457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:51.2626964 +0000 UTC m=+1337.179352430" watchObservedRunningTime="2026-09-10 19:32:51.264886457 +0000 UTC m=+1337.181542518" Sep 10 19:32:54.673211 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:32:54.673152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:02.254958 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:02.254850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:33:08.359311 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:08.359273 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerID="c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b" exitCode=0 Sep 10 19:33:08.359874 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:08.359352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" event={"ID":"fd40f507-4b0a-410f-a820-ef1bd00566ee","Type":"ContainerDied","Data":"c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b"} Sep 10 19:33:08.359946 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:08.359929 2576 scope.go:117] "RemoveContainer" containerID="c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b" Sep 10 19:33:08.361220 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:08.361194 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:33:08.361280 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:08.361264 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:33:08.361317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:08.361297 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:33:09.173160 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:09.173125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:09.273525 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:09.273489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:09.368471 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:09.368431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" event={"ID":"fd40f507-4b0a-410f-a820-ef1bd00566ee","Type":"ContainerStarted","Data":"74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27"} Sep 10 19:33:09.579499 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:09.579451 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:33:09.579757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:09.579737 2576 patch_prober.go:28] interesting pod/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 10 19:33:09.579816 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:09.579794 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 10 19:33:10.463941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:10.463904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:10.580100 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:10.580033 2576 patch_prober.go:28] interesting pod/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 10 19:33:10.580274 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:10.580121 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 10 19:33:11.579834 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:11.579794 2576 patch_prober.go:28] interesting pod/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 10 19:33:11.580273 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:11.579855 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 10 19:33:12.580508 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:12.580465 2576 patch_prober.go:28] interesting pod/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 10 19:33:12.580967 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:12.580535 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 10 19:33:13.066561 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:13.066528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:13.580656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:13.580605 2576 patch_prober.go:28] interesting pod/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 10 19:33:13.581166 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:13.580702 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 10 19:33:14.788720 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:14.788688 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:33:14.789490 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:14.789438 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:33:14.790800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:14.790774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:33:15.888466 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:15.888411 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8"] Sep 10 19:33:15.892559 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:15.892533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" Sep 10 19:33:15.895456 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:15.895432 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:33:15.895609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:15.895440 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:33:15.901158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:15.901126 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8"] Sep 10 19:33:15.960541 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:15.960501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a59d37-0b37-45b5-ba5a-84d6186fcc75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8\" (UID: \"16a59d37-0b37-45b5-ba5a-84d6186fcc75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" Sep 10 19:33:16.049206 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:16.049169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:16.061856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:16.061821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a59d37-0b37-45b5-ba5a-84d6186fcc75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8\" (UID: \"16a59d37-0b37-45b5-ba5a-84d6186fcc75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" Sep 10 19:33:16.062535 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:16.062514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a59d37-0b37-45b5-ba5a-84d6186fcc75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8\" (UID: \"16a59d37-0b37-45b5-ba5a-84d6186fcc75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" Sep 10 19:33:16.207246 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:16.207152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" Sep 10 19:33:16.483538 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:16.483504 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8"] Sep 10 19:33:16.485222 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:33:16.485196 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a59d37_0b37_45b5_ba5a_84d6186fcc75.slice/crio-fd4f409cbe5d85d152dd820492ee047ae2a8c9e5675242ef7a57b747c28498f5 WatchSource:0}: Error finding container fd4f409cbe5d85d152dd820492ee047ae2a8c9e5675242ef7a57b747c28498f5: Status 404 returned error can't find the container with id fd4f409cbe5d85d152dd820492ee047ae2a8c9e5675242ef7a57b747c28498f5 Sep 10 19:33:17.425134 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:17.425030 2576 generic.go:356] "Generic (PLEG): container finished" podID="16a59d37-0b37-45b5-ba5a-84d6186fcc75" containerID="7554edede140ef4932bda4ec78050ff5dc241bed0a39e572cbbcc68b37c02d9e" exitCode=0 Sep 10 19:33:17.425134 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:17.425071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" event={"ID":"16a59d37-0b37-45b5-ba5a-84d6186fcc75","Type":"ContainerDied","Data":"7554edede140ef4932bda4ec78050ff5dc241bed0a39e572cbbcc68b37c02d9e"} Sep 10 19:33:17.425134 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:17.425129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" event={"ID":"16a59d37-0b37-45b5-ba5a-84d6186fcc75","Type":"ContainerStarted","Data":"fd4f409cbe5d85d152dd820492ee047ae2a8c9e5675242ef7a57b747c28498f5"} Sep 10 19:33:17.460879 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:17.460838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:18.577444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:18.577414 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" Sep 10 19:33:18.688540 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:18.688442 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a59d37-0b37-45b5-ba5a-84d6186fcc75-maas-api-service-ca\") pod \"16a59d37-0b37-45b5-ba5a-84d6186fcc75\" (UID: \"16a59d37-0b37-45b5-ba5a-84d6186fcc75\") " Sep 10 19:33:18.688843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:18.688818 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a59d37-0b37-45b5-ba5a-84d6186fcc75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16a59d37-0b37-45b5-ba5a-84d6186fcc75" (UID: "16a59d37-0b37-45b5-ba5a-84d6186fcc75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:33:18.789954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:18.789910 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a59d37-0b37-45b5-ba5a-84d6186fcc75-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:19.439487 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:19.439449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" event={"ID":"16a59d37-0b37-45b5-ba5a-84d6186fcc75","Type":"ContainerDied","Data":"fd4f409cbe5d85d152dd820492ee047ae2a8c9e5675242ef7a57b747c28498f5"} Sep 10 19:33:19.439487 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:19.439474 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8" Sep 10 19:33:19.439707 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:19.439500 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd4f409cbe5d85d152dd820492ee047ae2a8c9e5675242ef7a57b747c28498f5" Sep 10 19:33:25.704609 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:25.704575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:31.470606 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.470572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:31.503490 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.503459 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p"] Sep 10 19:33:31.503760 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.503732 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" podUID="f3c49716-dec6-4a3d-89b0-dcd4afc23344" containerName="payload-processing" containerID="cri-o://f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a" gracePeriod=30 Sep 10 19:33:31.513717 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.513571 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72"] Sep 10 19:33:31.514002 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.513805 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" podUID="b9da5046-8fbe-439b-b221-cabeed0fb37c" containerName="payload-pre-processing" containerID="cri-o://001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d" gracePeriod=30 Sep 10 19:33:31.548031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.547995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:31.623466 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.623420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:31.712301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.712255 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8"] Sep 10 19:33:31.714234 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.714203 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd7dca-l68x8"] Sep 10 19:33:31.716238 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.716200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:31.788325 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.788294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:31.824216 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.824186 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:33:31.832671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.832632 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:33:31.925724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.925685 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9da5046-8fbe-439b-b221-cabeed0fb37c-plugins-config-volume\") pod \"b9da5046-8fbe-439b-b221-cabeed0fb37c\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " Sep 10 19:33:31.925724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.925730 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c49716-dec6-4a3d-89b0-dcd4afc23344-plugins-config-volume\") pod \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " Sep 10 19:33:31.925956 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.925785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n9fm\" (UniqueName: \"kubernetes.io/projected/b9da5046-8fbe-439b-b221-cabeed0fb37c-kube-api-access-8n9fm\") pod \"b9da5046-8fbe-439b-b221-cabeed0fb37c\" (UID: \"b9da5046-8fbe-439b-b221-cabeed0fb37c\") " Sep 10 19:33:31.925956 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.925826 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5776v\" (UniqueName: \"kubernetes.io/projected/f3c49716-dec6-4a3d-89b0-dcd4afc23344-kube-api-access-5776v\") pod \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\" (UID: \"f3c49716-dec6-4a3d-89b0-dcd4afc23344\") " Sep 10 19:33:31.926105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.926081 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c49716-dec6-4a3d-89b0-dcd4afc23344-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3c49716-dec6-4a3d-89b0-dcd4afc23344" (UID: "f3c49716-dec6-4a3d-89b0-dcd4afc23344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:33:31.926164 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.926146 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9da5046-8fbe-439b-b221-cabeed0fb37c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9da5046-8fbe-439b-b221-cabeed0fb37c" (UID: "b9da5046-8fbe-439b-b221-cabeed0fb37c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:33:31.926214 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.926199 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c49716-dec6-4a3d-89b0-dcd4afc23344-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:31.928192 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.928166 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9da5046-8fbe-439b-b221-cabeed0fb37c-kube-api-access-8n9fm" (OuterVolumeSpecName: "kube-api-access-8n9fm") pod "b9da5046-8fbe-439b-b221-cabeed0fb37c" (UID: "b9da5046-8fbe-439b-b221-cabeed0fb37c"). InnerVolumeSpecName "kube-api-access-8n9fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:33:31.928312 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:31.928189 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c49716-dec6-4a3d-89b0-dcd4afc23344-kube-api-access-5776v" (OuterVolumeSpecName: "kube-api-access-5776v") pod "f3c49716-dec6-4a3d-89b0-dcd4afc23344" (UID: "f3c49716-dec6-4a3d-89b0-dcd4afc23344"). InnerVolumeSpecName "kube-api-access-5776v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:33:32.027110 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.027015 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9fm\" (UniqueName: \"kubernetes.io/projected/b9da5046-8fbe-439b-b221-cabeed0fb37c-kube-api-access-8n9fm\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:32.027110 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.027053 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5776v\" (UniqueName: \"kubernetes.io/projected/f3c49716-dec6-4a3d-89b0-dcd4afc23344-kube-api-access-5776v\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:32.027110 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.027063 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9da5046-8fbe-439b-b221-cabeed0fb37c-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:32.087477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.087439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:32.529058 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.529027 2576 generic.go:356] "Generic (PLEG): container finished" podID="f3c49716-dec6-4a3d-89b0-dcd4afc23344" containerID="f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a" exitCode=0 Sep 10 19:33:32.529590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.529076 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" Sep 10 19:33:32.529590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.529102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" event={"ID":"f3c49716-dec6-4a3d-89b0-dcd4afc23344","Type":"ContainerDied","Data":"f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a"} Sep 10 19:33:32.529590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.529193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p" event={"ID":"f3c49716-dec6-4a3d-89b0-dcd4afc23344","Type":"ContainerDied","Data":"e016f0e7a2adf5755941aeec455f72037ba95b611992c6f66f175e533ae9fdbf"} Sep 10 19:33:32.529590 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.529229 2576 scope.go:117] "RemoveContainer" containerID="f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a" Sep 10 19:33:32.530757 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.530731 2576 generic.go:356] "Generic (PLEG): container finished" podID="b9da5046-8fbe-439b-b221-cabeed0fb37c" containerID="001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d" exitCode=0 Sep 10 19:33:32.530872 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.530775 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" Sep 10 19:33:32.530928 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.530838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" event={"ID":"b9da5046-8fbe-439b-b221-cabeed0fb37c","Type":"ContainerDied","Data":"001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d"} Sep 10 19:33:32.530928 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.530917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72" event={"ID":"b9da5046-8fbe-439b-b221-cabeed0fb37c","Type":"ContainerDied","Data":"f11af8c83ce026a107ca44d5c779619c87ccc337255699b861b8c54f77d9de9e"} Sep 10 19:33:32.540353 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.540330 2576 scope.go:117] "RemoveContainer" containerID="f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a" Sep 10 19:33:32.540718 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:33:32.540699 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a\": container with ID starting with f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a not found: ID does not exist" containerID="f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a" Sep 10 19:33:32.540836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.540728 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a"} err="failed to get container status \"f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a\": rpc error: code = NotFound desc = could not find container \"f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a\": container with ID starting with f00e98b3e05d8bb971d8429302bc824459b60767771a8a56c7298336e0e4069a not found: ID does not exist" Sep 10 19:33:32.540836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.540746 2576 scope.go:117] "RemoveContainer" containerID="001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d" Sep 10 19:33:32.550331 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.550310 2576 scope.go:117] "RemoveContainer" containerID="001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d" Sep 10 19:33:32.550686 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:33:32.550666 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d\": container with ID starting with 001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d not found: ID does not exist" containerID="001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d" Sep 10 19:33:32.550758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.550695 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d"} err="failed to get container status \"001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d\": rpc error: code = NotFound desc = could not find container \"001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d\": container with ID starting with 001af24cd0b7bc6b9dcd460f2e7e1da1fd591446e651825df0251a9b9535548d not found: ID does not exist" Sep 10 19:33:32.558182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.558156 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72"] Sep 10 19:33:32.565218 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.565193 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd7dca-b778b67cd-mwr72"] Sep 10 19:33:32.579764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.579732 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p"] Sep 10 19:33:32.583454 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.583431 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd7dca-7c854b78cf-k2n4p"] Sep 10 19:33:32.602548 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.602519 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16a59d37-0b37-45b5-ba5a-84d6186fcc75" path="/var/lib/kubelet/pods/16a59d37-0b37-45b5-ba5a-84d6186fcc75/volumes" Sep 10 19:33:32.602911 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.602899 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9da5046-8fbe-439b-b221-cabeed0fb37c" path="/var/lib/kubelet/pods/b9da5046-8fbe-439b-b221-cabeed0fb37c/volumes" Sep 10 19:33:32.603213 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:32.603201 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c49716-dec6-4a3d-89b0-dcd4afc23344" path="/var/lib/kubelet/pods/f3c49716-dec6-4a3d-89b0-dcd4afc23344/volumes" Sep 10 19:33:35.642331 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:35.642297 2576 scope.go:117] "RemoveContainer" containerID="45935b06eba9e0d7282a6356090c66de8f45627d5d68628ce703ba69c5411cdc" Sep 10 19:33:38.383059 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:38.383009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:38.497451 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:38.497420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:38.708244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:38.708154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:39.019412 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019313 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp"] Sep 10 19:33:39.019769 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019754 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c49716-dec6-4a3d-89b0-dcd4afc23344" containerName="payload-processing" Sep 10 19:33:39.019820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019771 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c49716-dec6-4a3d-89b0-dcd4afc23344" containerName="payload-processing" Sep 10 19:33:39.019820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019794 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9da5046-8fbe-439b-b221-cabeed0fb37c" containerName="payload-pre-processing" Sep 10 19:33:39.019820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019800 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9da5046-8fbe-439b-b221-cabeed0fb37c" containerName="payload-pre-processing" Sep 10 19:33:39.019820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019807 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a59d37-0b37-45b5-ba5a-84d6186fcc75" containerName="revoke-keys" Sep 10 19:33:39.019820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019813 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a59d37-0b37-45b5-ba5a-84d6186fcc75" containerName="revoke-keys" Sep 10 19:33:39.020022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019869 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a59d37-0b37-45b5-ba5a-84d6186fcc75" containerName="revoke-keys" Sep 10 19:33:39.020022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019876 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9da5046-8fbe-439b-b221-cabeed0fb37c" containerName="payload-pre-processing" Sep 10 19:33:39.020022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.019884 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c49716-dec6-4a3d-89b0-dcd4afc23344" containerName="payload-processing" Sep 10 19:33:39.024695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.024674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" Sep 10 19:33:39.027464 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.027443 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:33:39.027665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.027463 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:33:39.035102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.035068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp"] Sep 10 19:33:39.096131 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.096095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/608e6063-b971-4107-b992-4b3526b3f685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp\" (UID: \"608e6063-b971-4107-b992-4b3526b3f685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" Sep 10 19:33:39.196821 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.196784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/608e6063-b971-4107-b992-4b3526b3f685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp\" (UID: \"608e6063-b971-4107-b992-4b3526b3f685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" Sep 10 19:33:39.197436 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.197416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/608e6063-b971-4107-b992-4b3526b3f685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp\" (UID: \"608e6063-b971-4107-b992-4b3526b3f685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" Sep 10 19:33:39.256036 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.255996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:39.337403 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.337337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" Sep 10 19:33:39.616276 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.616251 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp"] Sep 10 19:33:39.617994 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:33:39.617957 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608e6063_b971_4107_b992_4b3526b3f685.slice/crio-7ee42a533ec647dce048f1278b4a9570e59fc015cc5ccce326af26dab5003858 WatchSource:0}: Error finding container 7ee42a533ec647dce048f1278b4a9570e59fc015cc5ccce326af26dab5003858: Status 404 returned error can't find the container with id 7ee42a533ec647dce048f1278b4a9570e59fc015cc5ccce326af26dab5003858 Sep 10 19:33:39.703116 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:39.703085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:40.584866 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:40.584831 2576 generic.go:356] "Generic (PLEG): container finished" podID="608e6063-b971-4107-b992-4b3526b3f685" containerID="d62a9ffb0f1c11c76e44dcd88f497ed13d02a7505fb93c7ebba07db2af25a001" exitCode=0 Sep 10 19:33:40.585048 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:40.584901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" event={"ID":"608e6063-b971-4107-b992-4b3526b3f685","Type":"ContainerDied","Data":"d62a9ffb0f1c11c76e44dcd88f497ed13d02a7505fb93c7ebba07db2af25a001"} Sep 10 19:33:40.585048 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:40.584939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" event={"ID":"608e6063-b971-4107-b992-4b3526b3f685","Type":"ContainerStarted","Data":"7ee42a533ec647dce048f1278b4a9570e59fc015cc5ccce326af26dab5003858"} Sep 10 19:33:40.648907 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:40.648815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:41.739642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:41.739609 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" Sep 10 19:33:41.823363 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:41.823329 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/608e6063-b971-4107-b992-4b3526b3f685-maas-api-service-ca\") pod \"608e6063-b971-4107-b992-4b3526b3f685\" (UID: \"608e6063-b971-4107-b992-4b3526b3f685\") " Sep 10 19:33:41.823712 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:41.823685 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608e6063-b971-4107-b992-4b3526b3f685-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "608e6063-b971-4107-b992-4b3526b3f685" (UID: "608e6063-b971-4107-b992-4b3526b3f685"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:33:41.924362 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:41.924271 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/608e6063-b971-4107-b992-4b3526b3f685-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:42.598923 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:42.598892 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" Sep 10 19:33:42.602743 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:42.602712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp" event={"ID":"608e6063-b971-4107-b992-4b3526b3f685","Type":"ContainerDied","Data":"7ee42a533ec647dce048f1278b4a9570e59fc015cc5ccce326af26dab5003858"} Sep 10 19:33:42.602904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:42.602766 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ee42a533ec647dce048f1278b4a9570e59fc015cc5ccce326af26dab5003858" Sep 10 19:33:45.952089 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:45.952051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:49.224429 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:49.224359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:53.350634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:53.350601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:54.866894 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:54.866848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:54.907858 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:54.907822 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt"] Sep 10 19:33:54.908124 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:54.908094 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" podUID="4e951860-59a2-4953-adb4-df8ef7f8d959" containerName="payload-processing" containerID="cri-o://4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37" gracePeriod=30 Sep 10 19:33:54.968878 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:54.968838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:55.072073 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.072041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:55.109049 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.108893 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp"] Sep 10 19:33:55.115820 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.115720 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66e9b4-bwkqp"] Sep 10 19:33:55.168057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.168021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:55.179948 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.179922 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:33:55.250739 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.250708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpqlf\" (UniqueName: \"kubernetes.io/projected/4e951860-59a2-4953-adb4-df8ef7f8d959-kube-api-access-fpqlf\") pod \"4e951860-59a2-4953-adb4-df8ef7f8d959\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " Sep 10 19:33:55.250956 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.250753 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e951860-59a2-4953-adb4-df8ef7f8d959-plugins-config-volume\") pod \"4e951860-59a2-4953-adb4-df8ef7f8d959\" (UID: \"4e951860-59a2-4953-adb4-df8ef7f8d959\") " Sep 10 19:33:55.251365 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.251216 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e951860-59a2-4953-adb4-df8ef7f8d959-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e951860-59a2-4953-adb4-df8ef7f8d959" (UID: "4e951860-59a2-4953-adb4-df8ef7f8d959"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:33:55.255720 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.255399 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e951860-59a2-4953-adb4-df8ef7f8d959-kube-api-access-fpqlf" (OuterVolumeSpecName: "kube-api-access-fpqlf") pod "4e951860-59a2-4953-adb4-df8ef7f8d959" (UID: "4e951860-59a2-4953-adb4-df8ef7f8d959"). InnerVolumeSpecName "kube-api-access-fpqlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:33:55.352342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.352298 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqlf\" (UniqueName: \"kubernetes.io/projected/4e951860-59a2-4953-adb4-df8ef7f8d959-kube-api-access-fpqlf\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:55.352342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.352330 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e951860-59a2-4953-adb4-df8ef7f8d959-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:33:55.583957 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.583920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:33:55.682436 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.682400 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e951860-59a2-4953-adb4-df8ef7f8d959" containerID="4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37" exitCode=0 Sep 10 19:33:55.682638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.682454 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" Sep 10 19:33:55.682638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.682491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" event={"ID":"4e951860-59a2-4953-adb4-df8ef7f8d959","Type":"ContainerDied","Data":"4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37"} Sep 10 19:33:55.682638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.682546 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt" event={"ID":"4e951860-59a2-4953-adb4-df8ef7f8d959","Type":"ContainerDied","Data":"1f4ce787c27ad4f8321d177d6e9c97ff6ed521a306da91460fed9f0037f2bf9c"} Sep 10 19:33:55.682638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.682568 2576 scope.go:117] "RemoveContainer" containerID="4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37" Sep 10 19:33:55.693215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.693192 2576 scope.go:117] "RemoveContainer" containerID="4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37" Sep 10 19:33:55.693548 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:33:55.693526 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37\": container with ID starting with 4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37 not found: ID does not exist" containerID="4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37" Sep 10 19:33:55.693618 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.693562 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37"} err="failed to get container status \"4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37\": rpc error: code = NotFound desc = could not find container \"4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37\": container with ID starting with 4def13ed0b44a52ec35a4be692e25ac84a5d7783f28f46724e492ea768ed0c37 not found: ID does not exist" Sep 10 19:33:55.717507 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.717473 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt"] Sep 10 19:33:55.737044 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:55.737013 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-66e9b4-74d84b8cc7-gvjdt"] Sep 10 19:33:56.603055 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:56.603020 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e951860-59a2-4953-adb4-df8ef7f8d959" path="/var/lib/kubelet/pods/4e951860-59a2-4953-adb4-df8ef7f8d959/volumes" Sep 10 19:33:56.603487 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:33:56.603363 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="608e6063-b971-4107-b992-4b3526b3f685" path="/var/lib/kubelet/pods/608e6063-b971-4107-b992-4b3526b3f685/volumes" Sep 10 19:34:02.387268 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:02.387226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:02.514848 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:02.514815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:02.678472 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:02.678387 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr"] Sep 10 19:34:02.678628 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:02.678599 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" containerID="cri-o://74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27" gracePeriod=30 Sep 10 19:34:02.727591 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:02.727555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:03.038474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.038440 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm"] Sep 10 19:34:03.038950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.038930 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e951860-59a2-4953-adb4-df8ef7f8d959" containerName="payload-processing" Sep 10 19:34:03.039050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.038953 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e951860-59a2-4953-adb4-df8ef7f8d959" containerName="payload-processing" Sep 10 19:34:03.039050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.038988 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608e6063-b971-4107-b992-4b3526b3f685" containerName="revoke-keys" Sep 10 19:34:03.039050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.038997 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="608e6063-b971-4107-b992-4b3526b3f685" containerName="revoke-keys" Sep 10 19:34:03.039215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.039094 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e951860-59a2-4953-adb4-df8ef7f8d959" containerName="payload-processing" Sep 10 19:34:03.039215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.039107 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="608e6063-b971-4107-b992-4b3526b3f685" containerName="revoke-keys" Sep 10 19:34:03.043659 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.043635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" Sep 10 19:34:03.046663 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.046635 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:34:03.047651 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.047631 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:34:03.056878 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.056834 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm"] Sep 10 19:34:03.131923 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.131876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm\" (UID: \"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" Sep 10 19:34:03.232982 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.232946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm\" (UID: \"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" Sep 10 19:34:03.233623 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.233603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm\" (UID: \"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" Sep 10 19:34:03.355790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.355697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" Sep 10 19:34:03.375252 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.372606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:03.654050 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:34:03.654020 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4faf1e5_c73e_4bcf_a3be_c7ea36b1f9bc.slice/crio-7177c6482ad708b8dc9a164fc025eff4e49c777b1eafbbd1ff4ef84041ee0f14 WatchSource:0}: Error finding container 7177c6482ad708b8dc9a164fc025eff4e49c777b1eafbbd1ff4ef84041ee0f14: Status 404 returned error can't find the container with id 7177c6482ad708b8dc9a164fc025eff4e49c777b1eafbbd1ff4ef84041ee0f14 Sep 10 19:34:03.654444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.654409 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm"] Sep 10 19:34:03.736954 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:03.736915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" event={"ID":"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc","Type":"ContainerStarted","Data":"7177c6482ad708b8dc9a164fc025eff4e49c777b1eafbbd1ff4ef84041ee0f14"} Sep 10 19:34:04.752028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:04.751928 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc" containerID="f6f2f6a2b27cc0d7301c78f26fd8ef79b81d2e0429cc73d6bea2e4a0e9bd4058" exitCode=0 Sep 10 19:34:04.752453 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:04.752024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" event={"ID":"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc","Type":"ContainerDied","Data":"f6f2f6a2b27cc0d7301c78f26fd8ef79b81d2e0429cc73d6bea2e4a0e9bd4058"} Sep 10 19:34:05.903112 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:05.903079 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" Sep 10 19:34:06.061638 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:06.061607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc-maas-api-service-ca\") pod \"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc\" (UID: \"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc\") " Sep 10 19:34:06.061966 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:06.061941 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc" (UID: "a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:34:06.162780 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:06.162737 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:06.765775 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:06.765741 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" Sep 10 19:34:06.765955 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:06.765777 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm" event={"ID":"a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc","Type":"ContainerDied","Data":"7177c6482ad708b8dc9a164fc025eff4e49c777b1eafbbd1ff4ef84041ee0f14"} Sep 10 19:34:06.765955 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:06.765827 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7177c6482ad708b8dc9a164fc025eff4e49c777b1eafbbd1ff4ef84041ee0f14" Sep 10 19:34:07.948772 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:07.948704 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:34:08.079846 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.079804 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-credential-socket\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.079863 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-certs\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.079891 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-socket\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.079961 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-podinfo\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080031 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.079984 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-envoy\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080056 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd40f507-4b0a-410f-a820-ef1bd00566ee-istiod-ca-cert\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080093 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-token\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080118 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-data\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080154 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:34:08.080255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080173 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftjbs\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-kube-api-access-ftjbs\") pod \"fd40f507-4b0a-410f-a820-ef1bd00566ee\" (UID: \"fd40f507-4b0a-410f-a820-ef1bd00566ee\") " Sep 10 19:34:08.080255 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080195 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:34:08.080603 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080282 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:34:08.080603 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080435 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:34:08.080603 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080493 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd40f507-4b0a-410f-a820-ef1bd00566ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:34:08.080764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080612 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd40f507-4b0a-410f-a820-ef1bd00566ee-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.080764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080641 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.080764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080654 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.080764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080667 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.080764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.080680 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.082596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.082571 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:34:08.082596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.082578 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:34:08.082748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.082611 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:34:08.082748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.082612 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-kube-api-access-ftjbs" (OuterVolumeSpecName: "kube-api-access-ftjbs") pod "fd40f507-4b0a-410f-a820-ef1bd00566ee" (UID: "fd40f507-4b0a-410f-a820-ef1bd00566ee"). InnerVolumeSpecName "kube-api-access-ftjbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:34:08.181495 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.181456 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.181495 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.181488 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.181495 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.181496 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.181495 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.181507 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftjbs\" (UniqueName: \"kubernetes.io/projected/fd40f507-4b0a-410f-a820-ef1bd00566ee-kube-api-access-ftjbs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:08.779975 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.779938 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerID="74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27" exitCode=0 Sep 10 19:34:08.780158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.779989 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" Sep 10 19:34:08.780158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.780028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" event={"ID":"fd40f507-4b0a-410f-a820-ef1bd00566ee","Type":"ContainerDied","Data":"74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27"} Sep 10 19:34:08.780158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.780086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr" event={"ID":"fd40f507-4b0a-410f-a820-ef1bd00566ee","Type":"ContainerDied","Data":"a3f3d84fe86f43c4b163ab2ea09deb71c00ff5044e8f2588f5f5f049d634f605"} Sep 10 19:34:08.780158 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.780113 2576 scope.go:117] "RemoveContainer" containerID="74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27" Sep 10 19:34:08.792840 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.792817 2576 scope.go:117] "RemoveContainer" containerID="c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b" Sep 10 19:34:08.799744 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.799710 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr"] Sep 10 19:34:08.803577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.803324 2576 scope.go:117] "RemoveContainer" containerID="74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27" Sep 10 19:34:08.803733 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:34:08.803709 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27\": container with ID starting with 74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27 not found: ID does not exist" containerID="74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27" Sep 10 19:34:08.803795 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.803743 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27"} err="failed to get container status \"74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27\": rpc error: code = NotFound desc = could not find container \"74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27\": container with ID starting with 74ce23a134f38fa2b367ffd8481b7deffd4f819c7173ba78cd4d572223beaf27 not found: ID does not exist" Sep 10 19:34:08.803795 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.803763 2576 scope.go:117] "RemoveContainer" containerID="c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b" Sep 10 19:34:08.804068 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:34:08.804045 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b\": container with ID starting with c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b not found: ID does not exist" containerID="c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b" Sep 10 19:34:08.804130 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.804075 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b"} err="failed to get container status \"c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b\": rpc error: code = NotFound desc = could not find container \"c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b\": container with ID starting with c1fce79c003b6e788113c8f09e8a0cf28f5b4cadc081cd5d1920bc43ec60220b not found: ID does not exist" Sep 10 19:34:08.804336 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:08.804316 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-66e9b4-openshift-default-56cf75d47d-s5wbr"] Sep 10 19:34:10.604345 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:10.604309 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" path="/var/lib/kubelet/pods/fd40f507-4b0a-410f-a820-ef1bd00566ee/volumes" Sep 10 19:34:13.683316 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.683282 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg"] Sep 10 19:34:13.687341 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.687308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:13.688022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.688000 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cfa578-29817810-kbklg"] Sep 10 19:34:13.716456 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.716422 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj"] Sep 10 19:34:13.716955 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.716919 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" podUID="592c00f4-ce09-44f5-be90-42c236292006" containerName="payload-pre-processing" containerID="cri-o://6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49" gracePeriod=30 Sep 10 19:34:13.784779 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.784741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:13.877334 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.877297 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm"] Sep 10 19:34:13.878854 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.878824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:13.880050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.880029 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cfa578-n6tjm"] Sep 10 19:34:13.937791 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.937667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:13.994128 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:13.994097 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:34:14.138497 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.138457 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2d2q\" (UniqueName: \"kubernetes.io/projected/592c00f4-ce09-44f5-be90-42c236292006-kube-api-access-m2d2q\") pod \"592c00f4-ce09-44f5-be90-42c236292006\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " Sep 10 19:34:14.138683 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.138507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592c00f4-ce09-44f5-be90-42c236292006-plugins-config-volume\") pod \"592c00f4-ce09-44f5-be90-42c236292006\" (UID: \"592c00f4-ce09-44f5-be90-42c236292006\") " Sep 10 19:34:14.138934 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.138912 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592c00f4-ce09-44f5-be90-42c236292006-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "592c00f4-ce09-44f5-be90-42c236292006" (UID: "592c00f4-ce09-44f5-be90-42c236292006"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:34:14.140860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.140829 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592c00f4-ce09-44f5-be90-42c236292006-kube-api-access-m2d2q" (OuterVolumeSpecName: "kube-api-access-m2d2q") pod "592c00f4-ce09-44f5-be90-42c236292006" (UID: "592c00f4-ce09-44f5-be90-42c236292006"). InnerVolumeSpecName "kube-api-access-m2d2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:34:14.236729 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.236647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:14.239873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.239846 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2d2q\" (UniqueName: \"kubernetes.io/projected/592c00f4-ce09-44f5-be90-42c236292006-kube-api-access-m2d2q\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:14.239873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.239872 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592c00f4-ce09-44f5-be90-42c236292006-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:14.604299 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.604265 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93dfe84f-62b4-4569-b673-22ed1419162d" path="/var/lib/kubelet/pods/93dfe84f-62b4-4569-b673-22ed1419162d/volumes" Sep 10 19:34:14.604619 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.604605 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc" path="/var/lib/kubelet/pods/a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc/volumes" Sep 10 19:34:14.820998 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.820961 2576 generic.go:356] "Generic (PLEG): container finished" podID="592c00f4-ce09-44f5-be90-42c236292006" containerID="6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49" exitCode=0 Sep 10 19:34:14.821544 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.821011 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" Sep 10 19:34:14.821544 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.821055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" event={"ID":"592c00f4-ce09-44f5-be90-42c236292006","Type":"ContainerDied","Data":"6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49"} Sep 10 19:34:14.821544 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.821120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj" event={"ID":"592c00f4-ce09-44f5-be90-42c236292006","Type":"ContainerDied","Data":"c55f993cc8cb11e5bb093f6154763790b18f6baf0f798026090482b5d2f9f463"} Sep 10 19:34:14.821544 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.821158 2576 scope.go:117] "RemoveContainer" containerID="6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49" Sep 10 19:34:14.837292 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.837263 2576 scope.go:117] "RemoveContainer" containerID="6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49" Sep 10 19:34:14.837703 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:34:14.837680 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49\": container with ID starting with 6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49 not found: ID does not exist" containerID="6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49" Sep 10 19:34:14.837784 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.837711 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49"} err="failed to get container status \"6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49\": rpc error: code = NotFound desc = could not find container \"6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49\": container with ID starting with 6bd1f3365fa13877e775f7e7b29b01ef7918f806d650b2c188bdee7320dfec49 not found: ID does not exist" Sep 10 19:34:14.844599 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.844560 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj"] Sep 10 19:34:14.848427 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:14.848398 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cfa578-b7fccb846-wsxcj"] Sep 10 19:34:16.602579 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:16.602544 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="592c00f4-ce09-44f5-be90-42c236292006" path="/var/lib/kubelet/pods/592c00f4-ce09-44f5-be90-42c236292006/volumes" Sep 10 19:34:20.943565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:20.943532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:21.042085 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.042050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:21.262862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.262769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:21.545902 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.545860 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n"] Sep 10 19:34:21.546292 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546279 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="592c00f4-ce09-44f5-be90-42c236292006" containerName="payload-pre-processing" Sep 10 19:34:21.546348 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546293 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="592c00f4-ce09-44f5-be90-42c236292006" containerName="payload-pre-processing" Sep 10 19:34:21.546348 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546302 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc" containerName="revoke-keys" Sep 10 19:34:21.546348 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546307 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc" containerName="revoke-keys" Sep 10 19:34:21.546348 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546322 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" Sep 10 19:34:21.546348 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546329 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" Sep 10 19:34:21.546348 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546339 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" Sep 10 19:34:21.546348 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546344 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" Sep 10 19:34:21.546633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546431 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4faf1e5-c73e-4bcf-a3be-c7ea36b1f9bc" containerName="revoke-keys" Sep 10 19:34:21.546633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546441 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" Sep 10 19:34:21.546633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546448 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd40f507-4b0a-410f-a820-ef1bd00566ee" containerName="istio-proxy" Sep 10 19:34:21.546633 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.546455 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="592c00f4-ce09-44f5-be90-42c236292006" containerName="payload-pre-processing" Sep 10 19:34:21.550996 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.550975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" Sep 10 19:34:21.553955 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.553871 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:34:21.554445 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.553877 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:34:21.558979 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.558950 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n"] Sep 10 19:34:21.611293 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.611245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c38dbf-717f-4a5a-af21-a41b37430856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n\" (UID: \"99c38dbf-717f-4a5a-af21-a41b37430856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" Sep 10 19:34:21.657941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.657907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:21.712249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.712204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c38dbf-717f-4a5a-af21-a41b37430856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n\" (UID: \"99c38dbf-717f-4a5a-af21-a41b37430856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" Sep 10 19:34:21.712893 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.712871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c38dbf-717f-4a5a-af21-a41b37430856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n\" (UID: \"99c38dbf-717f-4a5a-af21-a41b37430856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" Sep 10 19:34:21.863970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:21.863881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" Sep 10 19:34:22.138854 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:22.138814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n"] Sep 10 19:34:22.140181 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:34:22.140144 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c38dbf_717f_4a5a_af21_a41b37430856.slice/crio-f1e2cb69a681019125c2ff180c8700a6fb2c6e86026ca08653d5f3f6cff2ed00 WatchSource:0}: Error finding container f1e2cb69a681019125c2ff180c8700a6fb2c6e86026ca08653d5f3f6cff2ed00: Status 404 returned error can't find the container with id f1e2cb69a681019125c2ff180c8700a6fb2c6e86026ca08653d5f3f6cff2ed00 Sep 10 19:34:22.872933 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:22.872902 2576 generic.go:356] "Generic (PLEG): container finished" podID="99c38dbf-717f-4a5a-af21-a41b37430856" containerID="39e358e03c36ec9de676463e58c190b5b8b9843f0ca568f2c259dff5c942b0e6" exitCode=0 Sep 10 19:34:22.873115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:22.872958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" event={"ID":"99c38dbf-717f-4a5a-af21-a41b37430856","Type":"ContainerDied","Data":"39e358e03c36ec9de676463e58c190b5b8b9843f0ca568f2c259dff5c942b0e6"} Sep 10 19:34:22.873115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:22.873002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" event={"ID":"99c38dbf-717f-4a5a-af21-a41b37430856","Type":"ContainerStarted","Data":"f1e2cb69a681019125c2ff180c8700a6fb2c6e86026ca08653d5f3f6cff2ed00"} Sep 10 19:34:24.019726 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.019695 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" Sep 10 19:34:24.034693 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.034658 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c38dbf-717f-4a5a-af21-a41b37430856-maas-api-service-ca\") pod \"99c38dbf-717f-4a5a-af21-a41b37430856\" (UID: \"99c38dbf-717f-4a5a-af21-a41b37430856\") " Sep 10 19:34:24.035013 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.034990 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c38dbf-717f-4a5a-af21-a41b37430856-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99c38dbf-717f-4a5a-af21-a41b37430856" (UID: "99c38dbf-717f-4a5a-af21-a41b37430856"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:34:24.136415 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.136304 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c38dbf-717f-4a5a-af21-a41b37430856-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:24.333771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.333729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:24.886469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.886305 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" Sep 10 19:34:24.886636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.886331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n" event={"ID":"99c38dbf-717f-4a5a-af21-a41b37430856","Type":"ContainerDied","Data":"f1e2cb69a681019125c2ff180c8700a6fb2c6e86026ca08653d5f3f6cff2ed00"} Sep 10 19:34:24.886636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:24.886572 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e2cb69a681019125c2ff180c8700a6fb2c6e86026ca08653d5f3f6cff2ed00" Sep 10 19:34:30.758264 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:30.758194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:31.962682 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:31.962652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:32.990419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:32.990360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:33.029896 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.029862 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq"] Sep 10 19:34:33.037555 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.037525 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-94ca92-29817810-kqtnq"] Sep 10 19:34:33.104490 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.104456 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd"] Sep 10 19:34:33.104691 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.104669 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" podUID="e1ef0e1c-2af6-4867-b132-e1a873445171" containerName="payload-processing" containerID="cri-o://83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8" gracePeriod=30 Sep 10 19:34:33.226669 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.226636 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:33.360947 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.360919 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:34:33.425334 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.424466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ef0e1c-2af6-4867-b132-e1a873445171-plugins-config-volume\") pod \"e1ef0e1c-2af6-4867-b132-e1a873445171\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " Sep 10 19:34:33.425334 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.424598 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p5dq\" (UniqueName: \"kubernetes.io/projected/e1ef0e1c-2af6-4867-b132-e1a873445171-kube-api-access-5p5dq\") pod \"e1ef0e1c-2af6-4867-b132-e1a873445171\" (UID: \"e1ef0e1c-2af6-4867-b132-e1a873445171\") " Sep 10 19:34:33.425679 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.425535 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ef0e1c-2af6-4867-b132-e1a873445171-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1ef0e1c-2af6-4867-b132-e1a873445171" (UID: "e1ef0e1c-2af6-4867-b132-e1a873445171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:34:33.427974 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.427938 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ef0e1c-2af6-4867-b132-e1a873445171-kube-api-access-5p5dq" (OuterVolumeSpecName: "kube-api-access-5p5dq") pod "e1ef0e1c-2af6-4867-b132-e1a873445171" (UID: "e1ef0e1c-2af6-4867-b132-e1a873445171"). InnerVolumeSpecName "kube-api-access-5p5dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:34:33.509202 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.509165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:33.526110 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.526073 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ef0e1c-2af6-4867-b132-e1a873445171-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:33.526110 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.526111 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p5dq\" (UniqueName: \"kubernetes.io/projected/e1ef0e1c-2af6-4867-b132-e1a873445171-kube-api-access-5p5dq\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:34:33.596521 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.596484 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n"] Sep 10 19:34:33.600748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.600718 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-94ca92-5dp8n"] Sep 10 19:34:33.619697 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.619666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:33.947088 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.946990 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1ef0e1c-2af6-4867-b132-e1a873445171" containerID="83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8" exitCode=0 Sep 10 19:34:33.947088 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.947035 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" Sep 10 19:34:33.947312 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.947079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" event={"ID":"e1ef0e1c-2af6-4867-b132-e1a873445171","Type":"ContainerDied","Data":"83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8"} Sep 10 19:34:33.947312 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.947135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd" event={"ID":"e1ef0e1c-2af6-4867-b132-e1a873445171","Type":"ContainerDied","Data":"fe82d5cb57375762a7419aff241a560341b7f68744f30c46c68e64a0a6a637fa"} Sep 10 19:34:33.947312 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.947160 2576 scope.go:117] "RemoveContainer" containerID="83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8" Sep 10 19:34:33.957148 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.957129 2576 scope.go:117] "RemoveContainer" containerID="83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8" Sep 10 19:34:33.957457 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:34:33.957435 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8\": container with ID starting with 83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8 not found: ID does not exist" containerID="83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8" Sep 10 19:34:33.957519 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.957472 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8"} err="failed to get container status \"83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8\": rpc error: code = NotFound desc = could not find container \"83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8\": container with ID starting with 83bfae773bc30a74a942cd40fdcb0bdeed2f2c13648cd104e7453b37140214d8 not found: ID does not exist" Sep 10 19:34:33.986157 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.986121 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd"] Sep 10 19:34:33.993476 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:33.993444 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-94ca92-665f686f47-hbjnd"] Sep 10 19:34:34.012817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:34.012788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:34.602934 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:34.602903 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8f0063-5d60-4afd-86d5-489c600a623b" path="/var/lib/kubelet/pods/1e8f0063-5d60-4afd-86d5-489c600a623b/volumes" Sep 10 19:34:34.603200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:34.603188 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c38dbf-717f-4a5a-af21-a41b37430856" path="/var/lib/kubelet/pods/99c38dbf-717f-4a5a-af21-a41b37430856/volumes" Sep 10 19:34:34.603524 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:34.603511 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ef0e1c-2af6-4867-b132-e1a873445171" path="/var/lib/kubelet/pods/e1ef0e1c-2af6-4867-b132-e1a873445171/volumes" Sep 10 19:34:40.080764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:40.080728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:40.180941 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:40.180909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:40.340155 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:40.340055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:55.817192 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:55.817156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:34:59.983680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:34:59.983640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:00.058922 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:00.058894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:00.261153 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:00.261064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:00.606804 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:00.605473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:01.045615 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:01.045580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:03.404132 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:03.404088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:03.501621 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:03.501589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:04.662386 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:04.662331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:04.744827 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:04.744792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:05.167308 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.167272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:05.286752 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.286715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:05.490993 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.490898 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l"] Sep 10 19:35:05.491408 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.491364 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c38dbf-717f-4a5a-af21-a41b37430856" containerName="revoke-keys" Sep 10 19:35:05.491408 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.491403 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c38dbf-717f-4a5a-af21-a41b37430856" containerName="revoke-keys" Sep 10 19:35:05.491580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.491429 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ef0e1c-2af6-4867-b132-e1a873445171" containerName="payload-processing" Sep 10 19:35:05.491580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.491437 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ef0e1c-2af6-4867-b132-e1a873445171" containerName="payload-processing" Sep 10 19:35:05.491580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.491549 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ef0e1c-2af6-4867-b132-e1a873445171" containerName="payload-processing" Sep 10 19:35:05.491580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.491565 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c38dbf-717f-4a5a-af21-a41b37430856" containerName="revoke-keys" Sep 10 19:35:05.494764 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.494741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:05.498145 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.498097 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c59b36-dockercfg-fb7ds\"" Sep 10 19:35:05.498356 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.498337 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c59b36\"" Sep 10 19:35:05.507326 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.507299 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l"] Sep 10 19:35:05.519856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.519824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtx2z\" (UniqueName: \"kubernetes.io/projected/78c9d481-f1fa-40a6-84e3-9065e028474c-kube-api-access-gtx2z\") pod \"payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:05.520025 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.519879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c9d481-f1fa-40a6-84e3-9065e028474c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:05.530699 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.530657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:05.620663 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.620628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtx2z\" (UniqueName: \"kubernetes.io/projected/78c9d481-f1fa-40a6-84e3-9065e028474c-kube-api-access-gtx2z\") pod \"payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:05.620841 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.620686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c9d481-f1fa-40a6-84e3-9065e028474c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:05.621426 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.621401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c9d481-f1fa-40a6-84e3-9065e028474c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:05.629301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.629267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtx2z\" (UniqueName: \"kubernetes.io/projected/78c9d481-f1fa-40a6-84e3-9065e028474c-kube-api-access-gtx2z\") pod \"payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:05.806496 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:05.806460 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:06.085491 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:06.085465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l"] Sep 10 19:35:06.087908 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:35:06.087882 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c9d481_f1fa_40a6_84e3_9065e028474c.slice/crio-e8ed5104aaeae359e74fa2d7826b12c783599daf212d9dc16f4160e3c1e2d54f WatchSource:0}: Error finding container e8ed5104aaeae359e74fa2d7826b12c783599daf212d9dc16f4160e3c1e2d54f: Status 404 returned error can't find the container with id e8ed5104aaeae359e74fa2d7826b12c783599daf212d9dc16f4160e3c1e2d54f Sep 10 19:35:06.154521 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:06.154490 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" event={"ID":"78c9d481-f1fa-40a6-84e3-9065e028474c","Type":"ContainerStarted","Data":"e8ed5104aaeae359e74fa2d7826b12c783599daf212d9dc16f4160e3c1e2d54f"} Sep 10 19:35:07.162593 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:07.162551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" event={"ID":"78c9d481-f1fa-40a6-84e3-9065e028474c","Type":"ContainerStarted","Data":"5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c"} Sep 10 19:35:07.163003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:07.162635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:07.181350 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:07.181288 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" podStartSLOduration=2.181272119 podStartE2EDuration="2.181272119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:07.178249209 +0000 UTC m=+1473.094905239" watchObservedRunningTime="2026-09-10 19:35:07.181272119 +0000 UTC m=+1473.097928185" Sep 10 19:35:08.023566 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:08.023530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:18.175306 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:18.175270 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:35:28.396664 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:28.396628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:29.116843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:29.116784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:29.479451 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:29.479352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:29.690852 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:29.690808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:32.605520 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:32.605474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:32.699856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:32.699793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:33.759329 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:33.759294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:33.860834 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:33.860797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:34.304388 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.304334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:34.469054 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.469017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:34.562774 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.562644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:34.642802 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.642766 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9"] Sep 10 19:35:34.646503 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.646478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:34.649747 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.649722 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-90ceb3\"" Sep 10 19:35:34.650414 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.650394 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-90ceb3-dockercfg-jmqqp\"" Sep 10 19:35:34.662841 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.662804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9"] Sep 10 19:35:34.679400 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.679341 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:35:34.685129 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.685070 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:35:34.817517 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.817416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a59f08-6f95-4b5d-ba12-38f86dd01825-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:34.817517 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.817492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmcb\" (UniqueName: \"kubernetes.io/projected/31a59f08-6f95-4b5d-ba12-38f86dd01825-kube-api-access-6zmcb\") pod \"payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:34.919062 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.919018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a59f08-6f95-4b5d-ba12-38f86dd01825-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:34.919251 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.919093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmcb\" (UniqueName: \"kubernetes.io/projected/31a59f08-6f95-4b5d-ba12-38f86dd01825-kube-api-access-6zmcb\") pod \"payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:34.919877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.919855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a59f08-6f95-4b5d-ba12-38f86dd01825-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:34.931225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.931196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmcb\" (UniqueName: \"kubernetes.io/projected/31a59f08-6f95-4b5d-ba12-38f86dd01825-kube-api-access-6zmcb\") pod \"payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:34.960645 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:34.960605 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:35.244504 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:35.244477 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9"] Sep 10 19:35:35.246557 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:35:35.246525 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a59f08_6f95_4b5d_ba12_38f86dd01825.slice/crio-3d57342910e5ebc6f463affb7366dbe7f81ec3897527292c76c2f2e73d89ee01 WatchSource:0}: Error finding container 3d57342910e5ebc6f463affb7366dbe7f81ec3897527292c76c2f2e73d89ee01: Status 404 returned error can't find the container with id 3d57342910e5ebc6f463affb7366dbe7f81ec3897527292c76c2f2e73d89ee01 Sep 10 19:35:35.352486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:35.352433 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" event={"ID":"31a59f08-6f95-4b5d-ba12-38f86dd01825","Type":"ContainerStarted","Data":"29fe782b3b067aa879949977d6b6e98cf81c87f5936ca26e5d2f564d88fe8b5e"} Sep 10 19:35:35.352800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:35.352764 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:35.352927 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:35.352813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" event={"ID":"31a59f08-6f95-4b5d-ba12-38f86dd01825","Type":"ContainerStarted","Data":"3d57342910e5ebc6f463affb7366dbe7f81ec3897527292c76c2f2e73d89ee01"} Sep 10 19:35:35.384229 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:35.384153 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" podStartSLOduration=1.384132702 podStartE2EDuration="1.384132702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:35.384031016 +0000 UTC m=+1501.300687046" watchObservedRunningTime="2026-09-10 19:35:35.384132702 +0000 UTC m=+1501.300788734" Sep 10 19:35:35.702275 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:35.702245 2576 scope.go:117] "RemoveContainer" containerID="82b4c6dc6fab78b72952e148b99c136ba1386c595ce4eed88a397bf57187f2c9" Sep 10 19:35:46.361166 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:46.361130 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:35:52.604013 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:52.603982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:52.666261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:52.666228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:35:57.555816 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:35:57.555729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:02.528004 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:02.527970 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fbqwq_78633393-0b05-45f7-925c-2c41aceca7b4/payload-processing/0.log" Sep 10 19:36:09.569901 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:09.569867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:33.345203 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:33.345168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fbqwq_78633393-0b05-45f7-925c-2c41aceca7b4/payload-processing/0.log" Sep 10 19:36:33.750084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:33.749991 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:34.000346 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.000245 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd"] Sep 10 19:36:34.004366 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.004334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.007244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.007218 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-cbd9f0-openshift-default-dockercfg-q5r9c\"" Sep 10 19:36:34.018680 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.018647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd"] Sep 10 19:36:34.039322 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.039232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:34.105671 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-envoy\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.105877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4092b1f6-a37f-426a-a768-75c03ed1088c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.105877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-certs\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.105877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-credential-socket\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.105877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-podinfo\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.105877 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbdwv\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-kube-api-access-bbdwv\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.106113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-data\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.106113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.105988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-token\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.106113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.106014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-socket\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.206713 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-certs\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.206908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-credential-socket\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.206908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-podinfo\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.206908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbdwv\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-kube-api-access-bbdwv\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.206908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-data\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.206908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-token\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.206908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-socket\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.207244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-envoy\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.207244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.206956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4092b1f6-a37f-426a-a768-75c03ed1088c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.207244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.207139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-certs\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.207244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.207208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-data\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.207491 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.207326 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-socket\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.207491 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.207435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-credential-socket\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.207683 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.207666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4092b1f6-a37f-426a-a768-75c03ed1088c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.209364 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.209340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-envoy\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.209614 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.209595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-podinfo\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.215661 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.215641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbdwv\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-kube-api-access-bbdwv\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.215771 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.215751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-token\") pod \"e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.316057 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.316008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:34.440981 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.440952 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:34.595170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.595146 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd"] Sep 10 19:36:34.597096 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:36:34.597068 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4092b1f6_a37f_426a_a768_75c03ed1088c.slice/crio-a1ebb02fe25f675d90bbc6ac36ae5bfd96cb84771fe2cd3f1f71822dac9e3c3d WatchSource:0}: Error finding container a1ebb02fe25f675d90bbc6ac36ae5bfd96cb84771fe2cd3f1f71822dac9e3c3d: Status 404 returned error can't find the container with id a1ebb02fe25f675d90bbc6ac36ae5bfd96cb84771fe2cd3f1f71822dac9e3c3d Sep 10 19:36:34.599808 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.599766 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:36:34.599940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.599858 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:36:34.599940 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.599906 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:36:34.726616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.726573 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" event={"ID":"4092b1f6-a37f-426a-a768-75c03ed1088c","Type":"ContainerStarted","Data":"79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f"} Sep 10 19:36:34.726821 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.726635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" event={"ID":"4092b1f6-a37f-426a-a768-75c03ed1088c","Type":"ContainerStarted","Data":"a1ebb02fe25f675d90bbc6ac36ae5bfd96cb84771fe2cd3f1f71822dac9e3c3d"} Sep 10 19:36:34.748145 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:34.748079 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podStartSLOduration=1.748055469 podStartE2EDuration="1.748055469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:36:34.74648976 +0000 UTC m=+1560.663145790" watchObservedRunningTime="2026-09-10 19:36:34.748055469 +0000 UTC m=+1560.664711500" Sep 10 19:36:35.316692 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:35.316647 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:35.322549 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:35.322518 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:35.727591 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:35.727503 2576 scope.go:117] "RemoveContainer" containerID="37906a48607690418dabddd68aaa0ba422e5b4256c44a005a4411bcfd4d2940f" Sep 10 19:36:35.734068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:35.734036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:35.735063 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:35.735043 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:35.737696 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:35.737680 2576 scope.go:117] "RemoveContainer" containerID="bafe2a9b29a8b910f6ffc39301b4b7a5838833d8601870f2d42cfa800be19e68" Sep 10 19:36:35.746464 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:35.746445 2576 scope.go:117] "RemoveContainer" containerID="1ce2f831dcc3c10eca820cc4956a31a016894d6b9f0f437a0cbbbfabfc446595" Sep 10 19:36:36.644572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:36.644509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:36.735167 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:36.735137 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:37.813140 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:37.813110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:37.952594 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:37.952558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:38.054645 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.054615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:38.309022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.308986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:38.411718 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.411687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:38.573291 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.573209 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr"] Sep 10 19:36:38.579334 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.579297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:38.582250 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.582216 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-cbd9f0\"" Sep 10 19:36:38.582250 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.582230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-cbd9f0-dockercfg-rhg9k\"" Sep 10 19:36:38.598454 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.594609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr"] Sep 10 19:36:38.652124 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.652088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqb7s\" (UniqueName: \"kubernetes.io/projected/8bd03443-502c-41be-9695-605b064df69a-kube-api-access-qqb7s\") pod \"payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:38.652319 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.652142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd03443-502c-41be-9695-605b064df69a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:38.752746 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.752705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd03443-502c-41be-9695-605b064df69a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:38.752942 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.752812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqb7s\" (UniqueName: \"kubernetes.io/projected/8bd03443-502c-41be-9695-605b064df69a-kube-api-access-qqb7s\") pod \"payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:38.753446 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.753426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd03443-502c-41be-9695-605b064df69a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:38.765416 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.765366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqb7s\" (UniqueName: \"kubernetes.io/projected/8bd03443-502c-41be-9695-605b064df69a-kube-api-access-qqb7s\") pod \"payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:38.892884 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:38.892788 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:39.172502 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:39.172473 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr"] Sep 10 19:36:39.174792 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:36:39.174762 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd03443_502c_41be_9695_605b064df69a.slice/crio-f65452ba29b50efa5319d84b7fa87ade28254b5cc8049437f836cc267c33da62 WatchSource:0}: Error finding container f65452ba29b50efa5319d84b7fa87ade28254b5cc8049437f836cc267c33da62: Status 404 returned error can't find the container with id f65452ba29b50efa5319d84b7fa87ade28254b5cc8049437f836cc267c33da62 Sep 10 19:36:39.758724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:39.758687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" event={"ID":"8bd03443-502c-41be-9695-605b064df69a","Type":"ContainerStarted","Data":"f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d"} Sep 10 19:36:39.758974 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:39.758738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" event={"ID":"8bd03443-502c-41be-9695-605b064df69a","Type":"ContainerStarted","Data":"f65452ba29b50efa5319d84b7fa87ade28254b5cc8049437f836cc267c33da62"} Sep 10 19:36:39.758974 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:39.758776 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:39.775914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:39.775861 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" podStartSLOduration=1.775847191 podStartE2EDuration="1.775847191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:36:39.774922135 +0000 UTC m=+1565.691578166" watchObservedRunningTime="2026-09-10 19:36:39.775847191 +0000 UTC m=+1565.692503223" Sep 10 19:36:45.086522 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:45.086481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:36:50.766572 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:50.766540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:36:57.878352 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:57.878317 2576 generic.go:356] "Generic (PLEG): container finished" podID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerID="79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f" exitCode=0 Sep 10 19:36:57.878880 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:57.878410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" event={"ID":"4092b1f6-a37f-426a-a768-75c03ed1088c","Type":"ContainerDied","Data":"79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f"} Sep 10 19:36:57.878880 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:57.878864 2576 scope.go:117] "RemoveContainer" containerID="79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f" Sep 10 19:36:57.880020 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:57.879977 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:36:57.880122 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:57.880104 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:36:57.880188 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:57.880174 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 10 19:36:58.893738 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:58.893698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" event={"ID":"4092b1f6-a37f-426a-a768-75c03ed1088c","Type":"ContainerStarted","Data":"35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f"} Sep 10 19:36:59.317019 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:59.316986 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:36:59.317262 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:59.317237 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 19:36:59.317320 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:36:59.317299 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 19:37:00.316931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:00.316894 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 19:37:00.317450 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:00.316962 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 19:37:01.317015 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:01.316983 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 19:37:01.317417 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:01.317042 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 19:37:01.627087 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:01.626990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:02.288042 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.288000 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w"] Sep 10 19:37:02.292650 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.292625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" Sep 10 19:37:02.295421 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.295367 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:37:02.295597 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.295568 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:37:02.298191 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.298150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w"] Sep 10 19:37:02.317166 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.317131 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 19:37:02.317568 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.317191 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 19:37:02.390825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.390790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78105dee-fa72-4070-9608-8d9254a81218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w\" (UID: \"78105dee-fa72-4070-9608-8d9254a81218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" Sep 10 19:37:02.492615 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.492574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78105dee-fa72-4070-9608-8d9254a81218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w\" (UID: \"78105dee-fa72-4070-9608-8d9254a81218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" Sep 10 19:37:02.493299 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.493266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78105dee-fa72-4070-9608-8d9254a81218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w\" (UID: \"78105dee-fa72-4070-9608-8d9254a81218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" Sep 10 19:37:02.606653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.606574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" Sep 10 19:37:02.908109 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:37:02.908075 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78105dee_fa72_4070_9608_8d9254a81218.slice/crio-3d3c3117dea1bd7494c2013f9f39823c878bb166a4b9a0262eeec673adfefdeb WatchSource:0}: Error finding container 3d3c3117dea1bd7494c2013f9f39823c878bb166a4b9a0262eeec673adfefdeb: Status 404 returned error can't find the container with id 3d3c3117dea1bd7494c2013f9f39823c878bb166a4b9a0262eeec673adfefdeb Sep 10 19:37:02.909215 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.909189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w"] Sep 10 19:37:02.929096 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:02.929062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" event={"ID":"78105dee-fa72-4070-9608-8d9254a81218","Type":"ContainerStarted","Data":"3d3c3117dea1bd7494c2013f9f39823c878bb166a4b9a0262eeec673adfefdeb"} Sep 10 19:37:03.316428 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:03.316360 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 19:37:03.316626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:03.316446 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 19:37:03.938673 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:03.938634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" event={"ID":"78105dee-fa72-4070-9608-8d9254a81218","Type":"ContainerStarted","Data":"d388059d9e48b37c9bacb40a3cf12b4edcaf0fb28d4f360ce79cd5d9881edfd4"} Sep 10 19:37:03.964393 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:03.958626 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" podStartSLOduration=1.958595211 podStartE2EDuration="1.958595211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:37:03.954511551 +0000 UTC m=+1589.871167591" watchObservedRunningTime="2026-09-10 19:37:03.958595211 +0000 UTC m=+1589.875251241" Sep 10 19:37:04.321474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:04.321440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:37:04.321886 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:04.321854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:37:04.322666 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:04.322644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:37:04.945931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:04.945888 2576 generic.go:356] "Generic (PLEG): container finished" podID="78105dee-fa72-4070-9608-8d9254a81218" containerID="d388059d9e48b37c9bacb40a3cf12b4edcaf0fb28d4f360ce79cd5d9881edfd4" exitCode=0 Sep 10 19:37:04.946351 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:04.945957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" event={"ID":"78105dee-fa72-4070-9608-8d9254a81218","Type":"ContainerDied","Data":"d388059d9e48b37c9bacb40a3cf12b4edcaf0fb28d4f360ce79cd5d9881edfd4"} Sep 10 19:37:06.098116 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:06.098093 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" Sep 10 19:37:06.232630 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:06.232542 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78105dee-fa72-4070-9608-8d9254a81218-maas-api-service-ca\") pod \"78105dee-fa72-4070-9608-8d9254a81218\" (UID: \"78105dee-fa72-4070-9608-8d9254a81218\") " Sep 10 19:37:06.232914 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:06.232889 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78105dee-fa72-4070-9608-8d9254a81218-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78105dee-fa72-4070-9608-8d9254a81218" (UID: "78105dee-fa72-4070-9608-8d9254a81218"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:06.334235 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:06.334198 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78105dee-fa72-4070-9608-8d9254a81218-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:06.958812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:06.958722 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" Sep 10 19:37:06.958812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:06.958744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w" event={"ID":"78105dee-fa72-4070-9608-8d9254a81218","Type":"ContainerDied","Data":"3d3c3117dea1bd7494c2013f9f39823c878bb166a4b9a0262eeec673adfefdeb"} Sep 10 19:37:06.958812 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:06.958787 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d3c3117dea1bd7494c2013f9f39823c878bb166a4b9a0262eeec673adfefdeb" Sep 10 19:37:08.367817 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.367785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:08.402425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.402386 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr"] Sep 10 19:37:08.402704 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.402650 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" podUID="8bd03443-502c-41be-9695-605b064df69a" containerName="payload-processing" containerID="cri-o://f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d" gracePeriod=30 Sep 10 19:37:08.470918 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.470880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:08.566719 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.566690 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w"] Sep 10 19:37:08.568229 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.568190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:08.569532 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.569513 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cbd9f0-t446w"] Sep 10 19:37:08.614850 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.614813 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78105dee-fa72-4070-9608-8d9254a81218" path="/var/lib/kubelet/pods/78105dee-fa72-4070-9608-8d9254a81218/volumes" Sep 10 19:37:08.667983 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.667939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:08.683027 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.682995 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:37:08.760239 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.760201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqb7s\" (UniqueName: \"kubernetes.io/projected/8bd03443-502c-41be-9695-605b064df69a-kube-api-access-qqb7s\") pod \"8bd03443-502c-41be-9695-605b064df69a\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " Sep 10 19:37:08.760456 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.760333 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd03443-502c-41be-9695-605b064df69a-plugins-config-volume\") pod \"8bd03443-502c-41be-9695-605b064df69a\" (UID: \"8bd03443-502c-41be-9695-605b064df69a\") " Sep 10 19:37:08.760712 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.760688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd03443-502c-41be-9695-605b064df69a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bd03443-502c-41be-9695-605b064df69a" (UID: "8bd03443-502c-41be-9695-605b064df69a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:08.762584 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.762555 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd03443-502c-41be-9695-605b064df69a-kube-api-access-qqb7s" (OuterVolumeSpecName: "kube-api-access-qqb7s") pod "8bd03443-502c-41be-9695-605b064df69a" (UID: "8bd03443-502c-41be-9695-605b064df69a"). InnerVolumeSpecName "kube-api-access-qqb7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:08.861327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.861293 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqb7s\" (UniqueName: \"kubernetes.io/projected/8bd03443-502c-41be-9695-605b064df69a-kube-api-access-qqb7s\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:08.861327 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.861322 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd03443-502c-41be-9695-605b064df69a-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:08.943815 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.943727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:08.972893 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.972854 2576 generic.go:356] "Generic (PLEG): container finished" podID="8bd03443-502c-41be-9695-605b064df69a" containerID="f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d" exitCode=0 Sep 10 19:37:08.973084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.972899 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" Sep 10 19:37:08.973084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.972937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" event={"ID":"8bd03443-502c-41be-9695-605b064df69a","Type":"ContainerDied","Data":"f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d"} Sep 10 19:37:08.973084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.972989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr" event={"ID":"8bd03443-502c-41be-9695-605b064df69a","Type":"ContainerDied","Data":"f65452ba29b50efa5319d84b7fa87ade28254b5cc8049437f836cc267c33da62"} Sep 10 19:37:08.973084 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.973012 2576 scope.go:117] "RemoveContainer" containerID="f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d" Sep 10 19:37:08.986875 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.986846 2576 scope.go:117] "RemoveContainer" containerID="f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d" Sep 10 19:37:08.987200 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:37:08.987180 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d\": container with ID starting with f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d not found: ID does not exist" containerID="f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d" Sep 10 19:37:08.987273 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.987208 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d"} err="failed to get container status \"f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d\": rpc error: code = NotFound desc = could not find container \"f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d\": container with ID starting with f8ad07780464a1ead12886108f29fcbe56734346279ff1305d3919427341544d not found: ID does not exist" Sep 10 19:37:08.998085 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:08.998050 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr"] Sep 10 19:37:09.002060 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:09.002025 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cbd9f0-68fc7d846c-bt5lr"] Sep 10 19:37:10.603437 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:10.603404 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd03443-502c-41be-9695-605b064df69a" path="/var/lib/kubelet/pods/8bd03443-502c-41be-9695-605b064df69a/volumes" Sep 10 19:37:15.239726 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:15.239691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:15.332014 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:15.331983 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd"] Sep 10 19:37:15.332245 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:15.332209 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" containerID="cri-o://35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f" gracePeriod=30 Sep 10 19:37:15.348837 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:15.348790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:15.449065 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:15.449033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:18.872421 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.872361 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj"] Sep 10 19:37:18.872825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.872814 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd03443-502c-41be-9695-605b064df69a" containerName="payload-processing" Sep 10 19:37:18.872871 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.872827 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd03443-502c-41be-9695-605b064df69a" containerName="payload-processing" Sep 10 19:37:18.872871 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.872839 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78105dee-fa72-4070-9608-8d9254a81218" containerName="revoke-keys" Sep 10 19:37:18.872871 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.872844 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78105dee-fa72-4070-9608-8d9254a81218" containerName="revoke-keys" Sep 10 19:37:18.872964 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.872902 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd03443-502c-41be-9695-605b064df69a" containerName="payload-processing" Sep 10 19:37:18.872964 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.872910 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="78105dee-fa72-4070-9608-8d9254a81218" containerName="revoke-keys" Sep 10 19:37:18.877358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.877338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" Sep 10 19:37:18.880208 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.880181 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:37:18.880345 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.880326 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:37:18.885344 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.885317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj"] Sep 10 19:37:18.961879 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:18.961843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d30c45-7a07-453f-9ad4-0489c236cf52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj\" (UID: \"f0d30c45-7a07-453f-9ad4-0489c236cf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" Sep 10 19:37:19.063206 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:19.063164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d30c45-7a07-453f-9ad4-0489c236cf52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj\" (UID: \"f0d30c45-7a07-453f-9ad4-0489c236cf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" Sep 10 19:37:19.063832 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:19.063813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d30c45-7a07-453f-9ad4-0489c236cf52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj\" (UID: \"f0d30c45-7a07-453f-9ad4-0489c236cf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" Sep 10 19:37:19.145746 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:19.145635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:19.189822 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:19.189784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" Sep 10 19:37:19.323910 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:19.323870 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 19:37:19.324103 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:19.323933 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 19:37:19.473694 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:19.473665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj"] Sep 10 19:37:19.474804 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:37:19.474776 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d30c45_7a07_453f_9ad4_0489c236cf52.slice/crio-a73e67c82c6d9ad2ba1b739ed1d023f3906caa72a111917e90cdc1e697a1529a WatchSource:0}: Error finding container a73e67c82c6d9ad2ba1b739ed1d023f3906caa72a111917e90cdc1e697a1529a: Status 404 returned error can't find the container with id a73e67c82c6d9ad2ba1b739ed1d023f3906caa72a111917e90cdc1e697a1529a Sep 10 19:37:20.042076 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.042046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" event={"ID":"f0d30c45-7a07-453f-9ad4-0489c236cf52","Type":"ContainerStarted","Data":"394a7e8e9c33ebbe25757501426508a58bf46a87ecfbcbb18191a9e16e0fc539"} Sep 10 19:37:20.042542 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.042101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" event={"ID":"f0d30c45-7a07-453f-9ad4-0489c236cf52","Type":"ContainerStarted","Data":"a73e67c82c6d9ad2ba1b739ed1d023f3906caa72a111917e90cdc1e697a1529a"} Sep 10 19:37:20.586873 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.586844 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:37:20.609642 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.609522 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" podStartSLOduration=2.609500993 podStartE2EDuration="2.609500993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:37:20.063210652 +0000 UTC m=+1605.979866682" watchObservedRunningTime="2026-09-10 19:37:20.609500993 +0000 UTC m=+1606.526157025" Sep 10 19:37:20.678359 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678320 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-certs\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678359 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678360 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbdwv\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-kube-api-access-bbdwv\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678411 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-podinfo\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678439 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-data\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678462 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-token\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678492 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-credential-socket\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678636 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678547 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-socket\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678891 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-envoy\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678891 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678657 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:20.678891 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678690 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4092b1f6-a37f-426a-a768-75c03ed1088c-istiod-ca-cert\") pod \"4092b1f6-a37f-426a-a768-75c03ed1088c\" (UID: \"4092b1f6-a37f-426a-a768-75c03ed1088c\") " Sep 10 19:37:20.678891 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678846 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-data" (OuterVolumeSpecName: "istio-data") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:20.679106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678880 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:20.679106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.678939 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:20.679106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.679030 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.679106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.679050 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.679106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.679063 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.679106 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.679078 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.679598 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.679134 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4092b1f6-a37f-426a-a768-75c03ed1088c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:20.680818 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.680785 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-kube-api-access-bbdwv" (OuterVolumeSpecName: "kube-api-access-bbdwv") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "kube-api-access-bbdwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:20.680922 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.680891 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-token" (OuterVolumeSpecName: "istio-token") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:20.680995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.680976 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:37:20.681043 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.681019 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4092b1f6-a37f-426a-a768-75c03ed1088c" (UID: "4092b1f6-a37f-426a-a768-75c03ed1088c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:20.780425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.780353 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.780425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.780423 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4092b1f6-a37f-426a-a768-75c03ed1088c-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.780425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.780433 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbdwv\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-kube-api-access-bbdwv\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.780664 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.780443 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:20.780664 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:20.780451 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4092b1f6-a37f-426a-a768-75c03ed1088c-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:21.049477 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.049437 2576 generic.go:356] "Generic (PLEG): container finished" podID="f0d30c45-7a07-453f-9ad4-0489c236cf52" containerID="394a7e8e9c33ebbe25757501426508a58bf46a87ecfbcbb18191a9e16e0fc539" exitCode=0 Sep 10 19:37:21.049911 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.049519 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" event={"ID":"f0d30c45-7a07-453f-9ad4-0489c236cf52","Type":"ContainerDied","Data":"394a7e8e9c33ebbe25757501426508a58bf46a87ecfbcbb18191a9e16e0fc539"} Sep 10 19:37:21.051283 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.051257 2576 generic.go:356] "Generic (PLEG): container finished" podID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerID="35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f" exitCode=0 Sep 10 19:37:21.051425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.051301 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" Sep 10 19:37:21.051425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.051336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" event={"ID":"4092b1f6-a37f-426a-a768-75c03ed1088c","Type":"ContainerDied","Data":"35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f"} Sep 10 19:37:21.051425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.051407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd" event={"ID":"4092b1f6-a37f-426a-a768-75c03ed1088c","Type":"ContainerDied","Data":"a1ebb02fe25f675d90bbc6ac36ae5bfd96cb84771fe2cd3f1f71822dac9e3c3d"} Sep 10 19:37:21.051532 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.051434 2576 scope.go:117] "RemoveContainer" containerID="35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f" Sep 10 19:37:21.062187 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.062169 2576 scope.go:117] "RemoveContainer" containerID="79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f" Sep 10 19:37:21.071950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.071931 2576 scope.go:117] "RemoveContainer" containerID="35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f" Sep 10 19:37:21.072281 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:37:21.072256 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f\": container with ID starting with 35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f not found: ID does not exist" containerID="35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f" Sep 10 19:37:21.072337 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.072293 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f"} err="failed to get container status \"35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f\": rpc error: code = NotFound desc = could not find container \"35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f\": container with ID starting with 35afdbfd8d6bce22090fec51a63cb8c1c5a9df5a36555da82cf387b9292cca5f not found: ID does not exist" Sep 10 19:37:21.072337 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.072314 2576 scope.go:117] "RemoveContainer" containerID="79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f" Sep 10 19:37:21.072694 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:37:21.072673 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f\": container with ID starting with 79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f not found: ID does not exist" containerID="79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f" Sep 10 19:37:21.072770 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.072705 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f"} err="failed to get container status \"79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f\": rpc error: code = NotFound desc = could not find container \"79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f\": container with ID starting with 79a2c8223427dfa983f7e96febc36e29b2b20ae7225e7de3a028fdae6fd7836f not found: ID does not exist" Sep 10 19:37:21.080490 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.080465 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd"] Sep 10 19:37:21.085665 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:21.085639 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cbd9f0-openshift-default-847858ff56-bdvjd"] Sep 10 19:37:22.198338 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:22.198313 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" Sep 10 19:37:22.294679 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:22.294648 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d30c45-7a07-453f-9ad4-0489c236cf52-maas-api-service-ca\") pod \"f0d30c45-7a07-453f-9ad4-0489c236cf52\" (UID: \"f0d30c45-7a07-453f-9ad4-0489c236cf52\") " Sep 10 19:37:22.295050 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:22.295018 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d30c45-7a07-453f-9ad4-0489c236cf52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0d30c45-7a07-453f-9ad4-0489c236cf52" (UID: "f0d30c45-7a07-453f-9ad4-0489c236cf52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:22.395631 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:22.395548 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d30c45-7a07-453f-9ad4-0489c236cf52-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:22.603534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:22.603502 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" path="/var/lib/kubelet/pods/4092b1f6-a37f-426a-a768-75c03ed1088c/volumes" Sep 10 19:37:23.065755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:23.065726 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" Sep 10 19:37:23.065935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:23.065758 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj" event={"ID":"f0d30c45-7a07-453f-9ad4-0489c236cf52","Type":"ContainerDied","Data":"a73e67c82c6d9ad2ba1b739ed1d023f3906caa72a111917e90cdc1e697a1529a"} Sep 10 19:37:23.065935 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:23.065804 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a73e67c82c6d9ad2ba1b739ed1d023f3906caa72a111917e90cdc1e697a1529a" Sep 10 19:37:34.496461 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.496328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:34.533788 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.533753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l"] Sep 10 19:37:34.534028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.534003 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" podUID="78c9d481-f1fa-40a6-84e3-9065e028474c" containerName="payload-pre-processing" containerID="cri-o://5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c" gracePeriod=30 Sep 10 19:37:34.600136 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.600106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:34.697111 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.697070 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj"] Sep 10 19:37:34.699174 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.699113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:34.701487 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.701464 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c59b36-7zcxj"] Sep 10 19:37:34.798200 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.798139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:34.824425 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.824361 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:37:34.919097 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.919059 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtx2z\" (UniqueName: \"kubernetes.io/projected/78c9d481-f1fa-40a6-84e3-9065e028474c-kube-api-access-gtx2z\") pod \"78c9d481-f1fa-40a6-84e3-9065e028474c\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " Sep 10 19:37:34.919276 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.919207 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c9d481-f1fa-40a6-84e3-9065e028474c-plugins-config-volume\") pod \"78c9d481-f1fa-40a6-84e3-9065e028474c\" (UID: \"78c9d481-f1fa-40a6-84e3-9065e028474c\") " Sep 10 19:37:34.919602 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.919580 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c9d481-f1fa-40a6-84e3-9065e028474c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78c9d481-f1fa-40a6-84e3-9065e028474c" (UID: "78c9d481-f1fa-40a6-84e3-9065e028474c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:37:34.921385 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:34.921349 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c9d481-f1fa-40a6-84e3-9065e028474c-kube-api-access-gtx2z" (OuterVolumeSpecName: "kube-api-access-gtx2z") pod "78c9d481-f1fa-40a6-84e3-9065e028474c" (UID: "78c9d481-f1fa-40a6-84e3-9065e028474c"). InnerVolumeSpecName "kube-api-access-gtx2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:35.020360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.020317 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c9d481-f1fa-40a6-84e3-9065e028474c-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:35.020360 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.020352 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtx2z\" (UniqueName: \"kubernetes.io/projected/78c9d481-f1fa-40a6-84e3-9065e028474c-kube-api-access-gtx2z\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:37:35.122648 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.122556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:37:35.139658 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.139622 2576 generic.go:356] "Generic (PLEG): container finished" podID="78c9d481-f1fa-40a6-84e3-9065e028474c" containerID="5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c" exitCode=0 Sep 10 19:37:35.139835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.139671 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" Sep 10 19:37:35.139835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.139709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" event={"ID":"78c9d481-f1fa-40a6-84e3-9065e028474c","Type":"ContainerDied","Data":"5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c"} Sep 10 19:37:35.139835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.139772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l" event={"ID":"78c9d481-f1fa-40a6-84e3-9065e028474c","Type":"ContainerDied","Data":"e8ed5104aaeae359e74fa2d7826b12c783599daf212d9dc16f4160e3c1e2d54f"} Sep 10 19:37:35.139835 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.139805 2576 scope.go:117] "RemoveContainer" containerID="5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c" Sep 10 19:37:35.149752 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.149732 2576 scope.go:117] "RemoveContainer" containerID="5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c" Sep 10 19:37:35.150038 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:37:35.150018 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c\": container with ID starting with 5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c not found: ID does not exist" containerID="5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c" Sep 10 19:37:35.150090 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.150049 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c"} err="failed to get container status \"5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c\": rpc error: code = NotFound desc = could not find container \"5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c\": container with ID starting with 5abd307f35b7563b6098e52d4d4ba7d3a3d2669e86dbfed35888a4746520647c not found: ID does not exist" Sep 10 19:37:35.164244 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.164205 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l"] Sep 10 19:37:35.166667 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:35.166648 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c59b36-b654f6b6b-2g48l"] Sep 10 19:37:36.602956 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:36.602926 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c9d481-f1fa-40a6-84e3-9065e028474c" path="/var/lib/kubelet/pods/78c9d481-f1fa-40a6-84e3-9065e028474c/volumes" Sep 10 19:37:36.603342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:36.603320 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d30c45-7a07-453f-9ad4-0489c236cf52" path="/var/lib/kubelet/pods/f0d30c45-7a07-453f-9ad4-0489c236cf52/volumes" Sep 10 19:37:40.158300 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:37:40.158267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:16.692947 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:16.692912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:17.408288 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:17.408258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:18.212268 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:18.212234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:23.184004 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.183972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:23.255920 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.255879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:23.462860 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.462782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:23.802937 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.802898 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk"] Sep 10 19:38:23.803568 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803546 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c9d481-f1fa-40a6-84e3-9065e028474c" containerName="payload-pre-processing" Sep 10 19:38:23.803626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803573 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c9d481-f1fa-40a6-84e3-9065e028474c" containerName="payload-pre-processing" Sep 10 19:38:23.803626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803589 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d30c45-7a07-453f-9ad4-0489c236cf52" containerName="revoke-keys" Sep 10 19:38:23.803626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803598 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d30c45-7a07-453f-9ad4-0489c236cf52" containerName="revoke-keys" Sep 10 19:38:23.803626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803617 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" Sep 10 19:38:23.803626 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803626 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" Sep 10 19:38:23.803786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803642 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" Sep 10 19:38:23.803786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803649 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" Sep 10 19:38:23.803786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803741 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" Sep 10 19:38:23.803786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803753 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c9d481-f1fa-40a6-84e3-9065e028474c" containerName="payload-pre-processing" Sep 10 19:38:23.803786 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.803763 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d30c45-7a07-453f-9ad4-0489c236cf52" containerName="revoke-keys" Sep 10 19:38:23.807455 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.807436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" Sep 10 19:38:23.810212 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.810179 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:38:23.810468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.810446 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:38:23.812482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.812459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk"] Sep 10 19:38:23.872393 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.872331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3189e6-9c22-4c45-bbde-df3c719e0fea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk\" (UID: \"6a3189e6-9c22-4c45-bbde-df3c719e0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" Sep 10 19:38:23.973553 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.973506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3189e6-9c22-4c45-bbde-df3c719e0fea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk\" (UID: \"6a3189e6-9c22-4c45-bbde-df3c719e0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" Sep 10 19:38:23.974142 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:23.974120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3189e6-9c22-4c45-bbde-df3c719e0fea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk\" (UID: \"6a3189e6-9c22-4c45-bbde-df3c719e0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" Sep 10 19:38:24.088592 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:24.088490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:24.120706 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:24.120677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" Sep 10 19:38:24.394662 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:24.394636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk"] Sep 10 19:38:24.396055 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:38:24.396029 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3189e6_9c22_4c45_bbde_df3c719e0fea.slice/crio-ca21814db7aec7f0a2d0a385080999522966ea9e40328e2818a0a62cfbdc84fb WatchSource:0}: Error finding container ca21814db7aec7f0a2d0a385080999522966ea9e40328e2818a0a62cfbdc84fb: Status 404 returned error can't find the container with id ca21814db7aec7f0a2d0a385080999522966ea9e40328e2818a0a62cfbdc84fb Sep 10 19:38:24.438301 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:24.438275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" event={"ID":"6a3189e6-9c22-4c45-bbde-df3c719e0fea","Type":"ContainerStarted","Data":"ca21814db7aec7f0a2d0a385080999522966ea9e40328e2818a0a62cfbdc84fb"} Sep 10 19:38:25.446161 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:25.446068 2576 generic.go:356] "Generic (PLEG): container finished" podID="6a3189e6-9c22-4c45-bbde-df3c719e0fea" containerID="01ba9ea3c62eede22a99a04526b4ea21106cf83890b7d4b529179865b28aaa68" exitCode=0 Sep 10 19:38:25.446545 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:25.446173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" event={"ID":"6a3189e6-9c22-4c45-bbde-df3c719e0fea","Type":"ContainerDied","Data":"01ba9ea3c62eede22a99a04526b4ea21106cf83890b7d4b529179865b28aaa68"} Sep 10 19:38:26.586992 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:26.586961 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" Sep 10 19:38:26.700717 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:26.700679 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3189e6-9c22-4c45-bbde-df3c719e0fea-maas-api-service-ca\") pod \"6a3189e6-9c22-4c45-bbde-df3c719e0fea\" (UID: \"6a3189e6-9c22-4c45-bbde-df3c719e0fea\") " Sep 10 19:38:26.701052 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:26.701027 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3189e6-9c22-4c45-bbde-df3c719e0fea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a3189e6-9c22-4c45-bbde-df3c719e0fea" (UID: "6a3189e6-9c22-4c45-bbde-df3c719e0fea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:26.802157 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:26.802126 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3189e6-9c22-4c45-bbde-df3c719e0fea-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:38:27.461546 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:27.461512 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" Sep 10 19:38:27.461738 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:27.461550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk" event={"ID":"6a3189e6-9c22-4c45-bbde-df3c719e0fea","Type":"ContainerDied","Data":"ca21814db7aec7f0a2d0a385080999522966ea9e40328e2818a0a62cfbdc84fb"} Sep 10 19:38:27.461738 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:27.461605 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca21814db7aec7f0a2d0a385080999522966ea9e40328e2818a0a62cfbdc84fb" Sep 10 19:38:34.293079 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.293046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:34.343545 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.343494 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9"] Sep 10 19:38:34.343807 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.343779 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" podUID="31a59f08-6f95-4b5d-ba12-38f86dd01825" containerName="payload-pre-processing" containerID="cri-o://29fe782b3b067aa879949977d6b6e98cf81c87f5936ca26e5d2f564d88fe8b5e" gracePeriod=30 Sep 10 19:38:34.371143 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.371110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:34.455105 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.455069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:34.504911 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.504876 2576 generic.go:356] "Generic (PLEG): container finished" podID="31a59f08-6f95-4b5d-ba12-38f86dd01825" containerID="29fe782b3b067aa879949977d6b6e98cf81c87f5936ca26e5d2f564d88fe8b5e" exitCode=0 Sep 10 19:38:34.505068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.504943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" event={"ID":"31a59f08-6f95-4b5d-ba12-38f86dd01825","Type":"ContainerDied","Data":"29fe782b3b067aa879949977d6b6e98cf81c87f5936ca26e5d2f564d88fe8b5e"} Sep 10 19:38:34.545890 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.545795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:34.583908 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.583877 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk"] Sep 10 19:38:34.588699 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.588673 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90ceb3-l9tpk"] Sep 10 19:38:34.603318 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.603288 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a3189e6-9c22-4c45-bbde-df3c719e0fea" path="/var/lib/kubelet/pods/6a3189e6-9c22-4c45-bbde-df3c719e0fea/volumes" Sep 10 19:38:34.607506 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.607479 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:38:34.675201 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.675168 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a59f08-6f95-4b5d-ba12-38f86dd01825-plugins-config-volume\") pod \"31a59f08-6f95-4b5d-ba12-38f86dd01825\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " Sep 10 19:38:34.675396 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.675341 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zmcb\" (UniqueName: \"kubernetes.io/projected/31a59f08-6f95-4b5d-ba12-38f86dd01825-kube-api-access-6zmcb\") pod \"31a59f08-6f95-4b5d-ba12-38f86dd01825\" (UID: \"31a59f08-6f95-4b5d-ba12-38f86dd01825\") " Sep 10 19:38:34.675905 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.675643 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a59f08-6f95-4b5d-ba12-38f86dd01825-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31a59f08-6f95-4b5d-ba12-38f86dd01825" (UID: "31a59f08-6f95-4b5d-ba12-38f86dd01825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:34.675905 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.675763 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a59f08-6f95-4b5d-ba12-38f86dd01825-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:38:34.678267 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.678227 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a59f08-6f95-4b5d-ba12-38f86dd01825-kube-api-access-6zmcb" (OuterVolumeSpecName: "kube-api-access-6zmcb") pod "31a59f08-6f95-4b5d-ba12-38f86dd01825" (UID: "31a59f08-6f95-4b5d-ba12-38f86dd01825"). InnerVolumeSpecName "kube-api-access-6zmcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:38:34.777333 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.777289 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmcb\" (UniqueName: \"kubernetes.io/projected/31a59f08-6f95-4b5d-ba12-38f86dd01825-kube-api-access-6zmcb\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:38:34.936964 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:34.936843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:35.513061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:35.513028 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" Sep 10 19:38:35.513061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:35.513055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9" event={"ID":"31a59f08-6f95-4b5d-ba12-38f86dd01825","Type":"ContainerDied","Data":"3d57342910e5ebc6f463affb7366dbe7f81ec3897527292c76c2f2e73d89ee01"} Sep 10 19:38:35.513652 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:35.513145 2576 scope.go:117] "RemoveContainer" containerID="29fe782b3b067aa879949977d6b6e98cf81c87f5936ca26e5d2f564d88fe8b5e" Sep 10 19:38:35.543484 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:35.543454 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9"] Sep 10 19:38:35.545836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:35.545817 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90ceb3-884f49766-s79x9"] Sep 10 19:38:36.602012 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:36.601983 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a59f08-6f95-4b5d-ba12-38f86dd01825" path="/var/lib/kubelet/pods/31a59f08-6f95-4b5d-ba12-38f86dd01825/volumes" Sep 10 19:38:40.941478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:40.941413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:41.041758 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.041717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:41.200577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.200481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:41.562899 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.562865 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc"] Sep 10 19:38:41.563238 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.563227 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3189e6-9c22-4c45-bbde-df3c719e0fea" containerName="revoke-keys" Sep 10 19:38:41.563281 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.563240 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3189e6-9c22-4c45-bbde-df3c719e0fea" containerName="revoke-keys" Sep 10 19:38:41.563281 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.563260 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a59f08-6f95-4b5d-ba12-38f86dd01825" containerName="payload-pre-processing" Sep 10 19:38:41.563281 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.563265 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a59f08-6f95-4b5d-ba12-38f86dd01825" containerName="payload-pre-processing" Sep 10 19:38:41.563406 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.563329 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a59f08-6f95-4b5d-ba12-38f86dd01825" containerName="payload-pre-processing" Sep 10 19:38:41.563406 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.563337 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3189e6-9c22-4c45-bbde-df3c719e0fea" containerName="revoke-keys" Sep 10 19:38:41.563406 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.563346 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4092b1f6-a37f-426a-a768-75c03ed1088c" containerName="istio-proxy" Sep 10 19:38:41.567708 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.567685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" Sep 10 19:38:41.570931 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.570903 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:38:41.571587 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.571567 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:38:41.582482 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.582462 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc"] Sep 10 19:38:41.643183 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.643146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8376cc15-2ff5-4a02-a8d4-c35e4340ab03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc\" (UID: \"8376cc15-2ff5-4a02-a8d4-c35e4340ab03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" Sep 10 19:38:41.731142 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.731114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:41.744329 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.744296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8376cc15-2ff5-4a02-a8d4-c35e4340ab03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc\" (UID: \"8376cc15-2ff5-4a02-a8d4-c35e4340ab03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" Sep 10 19:38:41.745009 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.744991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8376cc15-2ff5-4a02-a8d4-c35e4340ab03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc\" (UID: \"8376cc15-2ff5-4a02-a8d4-c35e4340ab03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" Sep 10 19:38:41.878578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:41.878488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" Sep 10 19:38:42.161210 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:42.161185 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc"] Sep 10 19:38:42.162773 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:38:42.162748 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8376cc15_2ff5_4a02_a8d4_c35e4340ab03.slice/crio-f1115debdd36c43b43538a18d93bf116e7590e957da28216aaa1b713fc41f462 WatchSource:0}: Error finding container f1115debdd36c43b43538a18d93bf116e7590e957da28216aaa1b713fc41f462: Status 404 returned error can't find the container with id f1115debdd36c43b43538a18d93bf116e7590e957da28216aaa1b713fc41f462 Sep 10 19:38:42.559340 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:42.559305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" event={"ID":"8376cc15-2ff5-4a02-a8d4-c35e4340ab03","Type":"ContainerStarted","Data":"7963fadbcfa3d975a638cac99c9943e17f02a3eadf4c57be7e1ec7956c25dc97"} Sep 10 19:38:42.559340 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:42.559358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" event={"ID":"8376cc15-2ff5-4a02-a8d4-c35e4340ab03","Type":"ContainerStarted","Data":"f1115debdd36c43b43538a18d93bf116e7590e957da28216aaa1b713fc41f462"} Sep 10 19:38:42.588649 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:42.588597 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" podStartSLOduration=1.588579194 podStartE2EDuration="1.588579194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:38:42.587307206 +0000 UTC m=+1688.503963235" watchObservedRunningTime="2026-09-10 19:38:42.588579194 +0000 UTC m=+1688.505235222" Sep 10 19:38:43.567160 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:43.567126 2576 generic.go:356] "Generic (PLEG): container finished" podID="8376cc15-2ff5-4a02-a8d4-c35e4340ab03" containerID="7963fadbcfa3d975a638cac99c9943e17f02a3eadf4c57be7e1ec7956c25dc97" exitCode=0 Sep 10 19:38:43.567588 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:43.567189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" event={"ID":"8376cc15-2ff5-4a02-a8d4-c35e4340ab03","Type":"ContainerDied","Data":"7963fadbcfa3d975a638cac99c9943e17f02a3eadf4c57be7e1ec7956c25dc97"} Sep 10 19:38:44.729269 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:44.729244 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" Sep 10 19:38:44.874148 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:44.874033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8376cc15-2ff5-4a02-a8d4-c35e4340ab03-maas-api-service-ca\") pod \"8376cc15-2ff5-4a02-a8d4-c35e4340ab03\" (UID: \"8376cc15-2ff5-4a02-a8d4-c35e4340ab03\") " Sep 10 19:38:44.874438 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:44.874413 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8376cc15-2ff5-4a02-a8d4-c35e4340ab03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8376cc15-2ff5-4a02-a8d4-c35e4340ab03" (UID: "8376cc15-2ff5-4a02-a8d4-c35e4340ab03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:44.975496 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:44.975455 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8376cc15-2ff5-4a02-a8d4-c35e4340ab03-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:38:45.579686 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:45.579649 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" Sep 10 19:38:45.579686 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:45.579672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc" event={"ID":"8376cc15-2ff5-4a02-a8d4-c35e4340ab03","Type":"ContainerDied","Data":"f1115debdd36c43b43538a18d93bf116e7590e957da28216aaa1b713fc41f462"} Sep 10 19:38:45.579927 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:45.579757 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1115debdd36c43b43538a18d93bf116e7590e957da28216aaa1b713fc41f462" Sep 10 19:38:46.630997 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:46.630969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:51.691936 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:51.691898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:52.282233 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.282195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:52.286762 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.286732 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd"] Sep 10 19:38:52.289909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.289886 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b64b1c-29817810-lfvdd"] Sep 10 19:38:52.317061 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.317033 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z"] Sep 10 19:38:52.317289 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.317256 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" podUID="c490a42c-18b6-408a-baad-ce967460f21c" containerName="payload-processing" containerID="cri-o://1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493" gracePeriod=30 Sep 10 19:38:52.381992 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.381957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:52.481236 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.481204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:52.512243 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.512200 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc"] Sep 10 19:38:52.515176 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.515149 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b64b1c-htcgc"] Sep 10 19:38:52.573655 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.573602 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:38:52.603503 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.603471 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55935be2-3e6b-40d5-98fc-bb322a5fb2f9" path="/var/lib/kubelet/pods/55935be2-3e6b-40d5-98fc-bb322a5fb2f9/volumes" Sep 10 19:38:52.603888 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.603871 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8376cc15-2ff5-4a02-a8d4-c35e4340ab03" path="/var/lib/kubelet/pods/8376cc15-2ff5-4a02-a8d4-c35e4340ab03/volumes" Sep 10 19:38:52.622833 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.622805 2576 generic.go:356] "Generic (PLEG): container finished" podID="c490a42c-18b6-408a-baad-ce967460f21c" containerID="1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493" exitCode=0 Sep 10 19:38:52.623005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.622856 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" Sep 10 19:38:52.623005 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.622859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" event={"ID":"c490a42c-18b6-408a-baad-ce967460f21c","Type":"ContainerDied","Data":"1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493"} Sep 10 19:38:52.623103 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.622999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z" event={"ID":"c490a42c-18b6-408a-baad-ce967460f21c","Type":"ContainerDied","Data":"bdef74c43cbf1f69594593ec412916c7249547fa5b32b4ec402ff0474cd3a7a0"} Sep 10 19:38:52.623103 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.623023 2576 scope.go:117] "RemoveContainer" containerID="1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493" Sep 10 19:38:52.632622 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.632586 2576 scope.go:117] "RemoveContainer" containerID="1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493" Sep 10 19:38:52.632966 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:38:52.632941 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493\": container with ID starting with 1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493 not found: ID does not exist" containerID="1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493" Sep 10 19:38:52.633045 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.632977 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493"} err="failed to get container status \"1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493\": rpc error: code = NotFound desc = could not find container \"1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493\": container with ID starting with 1e02b81d9ed27a9546a23e7289b367c79dd2fbb2c179b7d5a89668432bea2493 not found: ID does not exist" Sep 10 19:38:52.653569 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.653540 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dbpr\" (UniqueName: \"kubernetes.io/projected/c490a42c-18b6-408a-baad-ce967460f21c-kube-api-access-2dbpr\") pod \"c490a42c-18b6-408a-baad-ce967460f21c\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " Sep 10 19:38:52.653742 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.653657 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c490a42c-18b6-408a-baad-ce967460f21c-plugins-config-volume\") pod \"c490a42c-18b6-408a-baad-ce967460f21c\" (UID: \"c490a42c-18b6-408a-baad-ce967460f21c\") " Sep 10 19:38:52.654028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.654007 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c490a42c-18b6-408a-baad-ce967460f21c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c490a42c-18b6-408a-baad-ce967460f21c" (UID: "c490a42c-18b6-408a-baad-ce967460f21c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:38:52.656670 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.656592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c490a42c-18b6-408a-baad-ce967460f21c-kube-api-access-2dbpr" (OuterVolumeSpecName: "kube-api-access-2dbpr") pod "c490a42c-18b6-408a-baad-ce967460f21c" (UID: "c490a42c-18b6-408a-baad-ce967460f21c"). InnerVolumeSpecName "kube-api-access-2dbpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:38:52.666212 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.666173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:52.754968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.754930 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c490a42c-18b6-408a-baad-ce967460f21c-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.754968 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.754968 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbpr\" (UniqueName: \"kubernetes.io/projected/c490a42c-18b6-408a-baad-ce967460f21c-kube-api-access-2dbpr\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:38:52.906020 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.905928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:52.947631 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.947601 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z"] Sep 10 19:38:52.957307 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:52.957267 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b64b1c-79949c789f-qg24z"] Sep 10 19:38:54.603037 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:54.603000 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c490a42c-18b6-408a-baad-ce967460f21c" path="/var/lib/kubelet/pods/c490a42c-18b6-408a-baad-ce967460f21c/volumes" Sep 10 19:38:59.463668 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:59.463575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:59.563631 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:59.563591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:38:59.782581 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:59.782494 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l"] Sep 10 19:38:59.782755 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:59.782710 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" containerID="cri-o://70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3" gracePeriod=30 Sep 10 19:38:59.826911 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:38:59.826872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:00.126297 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.126258 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt"] Sep 10 19:39:00.126695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.126681 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c490a42c-18b6-408a-baad-ce967460f21c" containerName="payload-processing" Sep 10 19:39:00.126748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.126697 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c490a42c-18b6-408a-baad-ce967460f21c" containerName="payload-processing" Sep 10 19:39:00.126748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.126720 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8376cc15-2ff5-4a02-a8d4-c35e4340ab03" containerName="revoke-keys" Sep 10 19:39:00.126748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.126726 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8376cc15-2ff5-4a02-a8d4-c35e4340ab03" containerName="revoke-keys" Sep 10 19:39:00.126850 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.126788 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c490a42c-18b6-408a-baad-ce967460f21c" containerName="payload-processing" Sep 10 19:39:00.126850 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.126801 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8376cc15-2ff5-4a02-a8d4-c35e4340ab03" containerName="revoke-keys" Sep 10 19:39:00.131522 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.131503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" Sep 10 19:39:00.135203 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.135176 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:39:00.136419 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.136396 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:39:00.140307 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.140273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt"] Sep 10 19:39:00.216949 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.216910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2609db0-1be7-4015-9c3f-9aa9a838b9e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt\" (UID: \"a2609db0-1be7-4015-9c3f-9aa9a838b9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" Sep 10 19:39:00.246974 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.246932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:00.317961 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.317922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2609db0-1be7-4015-9c3f-9aa9a838b9e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt\" (UID: \"a2609db0-1be7-4015-9c3f-9aa9a838b9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" Sep 10 19:39:00.318661 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.318638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2609db0-1be7-4015-9c3f-9aa9a838b9e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt\" (UID: \"a2609db0-1be7-4015-9c3f-9aa9a838b9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" Sep 10 19:39:00.445237 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.445146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" Sep 10 19:39:00.720486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:00.720459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt"] Sep 10 19:39:00.722527 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:39:00.722497 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2609db0_1be7_4015_9c3f_9aa9a838b9e8.slice/crio-61381bfc88f3a3ef002afce88b3075175c2c195e33d00e9e2bbc9e9d2c6f47eb WatchSource:0}: Error finding container 61381bfc88f3a3ef002afce88b3075175c2c195e33d00e9e2bbc9e9d2c6f47eb: Status 404 returned error can't find the container with id 61381bfc88f3a3ef002afce88b3075175c2c195e33d00e9e2bbc9e9d2c6f47eb Sep 10 19:39:01.307725 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:01.307637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:01.405116 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:01.405083 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 19:39:01.405287 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:01.405143 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 19:39:01.678695 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:01.678602 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2609db0-1be7-4015-9c3f-9aa9a838b9e8" containerID="e069b217dc42c2a82d192dea32753c6e1e9aebab2eb020c53ee962a2b2caa24c" exitCode=0 Sep 10 19:39:01.678844 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:01.678686 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" event={"ID":"a2609db0-1be7-4015-9c3f-9aa9a838b9e8","Type":"ContainerDied","Data":"e069b217dc42c2a82d192dea32753c6e1e9aebab2eb020c53ee962a2b2caa24c"} Sep 10 19:39:01.678844 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:01.678738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" event={"ID":"a2609db0-1be7-4015-9c3f-9aa9a838b9e8","Type":"ContainerStarted","Data":"61381bfc88f3a3ef002afce88b3075175c2c195e33d00e9e2bbc9e9d2c6f47eb"} Sep 10 19:39:02.817102 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:02.817075 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" Sep 10 19:39:02.841001 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:02.840967 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2609db0-1be7-4015-9c3f-9aa9a838b9e8-maas-api-service-ca\") pod \"a2609db0-1be7-4015-9c3f-9aa9a838b9e8\" (UID: \"a2609db0-1be7-4015-9c3f-9aa9a838b9e8\") " Sep 10 19:39:02.841435 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:02.841409 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2609db0-1be7-4015-9c3f-9aa9a838b9e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2609db0-1be7-4015-9c3f-9aa9a838b9e8" (UID: "a2609db0-1be7-4015-9c3f-9aa9a838b9e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:02.941709 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:02.941620 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2609db0-1be7-4015-9c3f-9aa9a838b9e8-maas-api-service-ca\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:03.691115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:03.691087 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" Sep 10 19:39:03.691302 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:03.691166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt" event={"ID":"a2609db0-1be7-4015-9c3f-9aa9a838b9e8","Type":"ContainerDied","Data":"61381bfc88f3a3ef002afce88b3075175c2c195e33d00e9e2bbc9e9d2c6f47eb"} Sep 10 19:39:03.691302 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:03.691217 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61381bfc88f3a3ef002afce88b3075175c2c195e33d00e9e2bbc9e9d2c6f47eb" Sep 10 19:39:05.038336 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.038305 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:39:05.060397 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060265 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-certs\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.060397 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060304 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-socket\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.060397 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060364 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01845e69-4217-4e5e-b6b7-c255f3325067-istio-podinfo\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.060811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060418 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbq45\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-kube-api-access-qbq45\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.060811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01845e69-4217-4e5e-b6b7-c255f3325067-istiod-ca-cert\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.060811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-data\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.060811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060619 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:39:05.060811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060759 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:39:05.061170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060940 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-credential-socket\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.061170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.060976 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-envoy\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.061170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.061050 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-istio-token\") pod \"01845e69-4217-4e5e-b6b7-c255f3325067\" (UID: \"01845e69-4217-4e5e-b6b7-c255f3325067\") " Sep 10 19:39:05.061170 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.061048 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01845e69-4217-4e5e-b6b7-c255f3325067-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:05.061456 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.061406 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.061456 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.061427 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-workload-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.062054 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.061731 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-data" (OuterVolumeSpecName: "istio-data") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:39:05.062054 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.062001 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:39:05.064203 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.064105 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01845e69-4217-4e5e-b6b7-c255f3325067-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 19:39:05.066339 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.066301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-istio-token" (OuterVolumeSpecName: "istio-token") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:05.066558 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.066529 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-kube-api-access-qbq45" (OuterVolumeSpecName: "kube-api-access-qbq45") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "kube-api-access-qbq45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:05.066644 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.066566 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "01845e69-4217-4e5e-b6b7-c255f3325067" (UID: "01845e69-4217-4e5e-b6b7-c255f3325067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:39:05.162596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.162558 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01845e69-4217-4e5e-b6b7-c255f3325067-istiod-ca-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.162596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.162590 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-data\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.162596 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.162600 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-credential-socket\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.162862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.162615 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01845e69-4217-4e5e-b6b7-c255f3325067-istio-envoy\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.162862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.162625 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-istio-token\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.162862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.162632 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01845e69-4217-4e5e-b6b7-c255f3325067-istio-podinfo\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.162862 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.162639 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbq45\" (UniqueName: \"kubernetes.io/projected/01845e69-4217-4e5e-b6b7-c255f3325067-kube-api-access-qbq45\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.704271 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.704237 2576 generic.go:356] "Generic (PLEG): container finished" podID="01845e69-4217-4e5e-b6b7-c255f3325067" containerID="70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3" exitCode=0 Sep 10 19:39:05.704469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.704292 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" Sep 10 19:39:05.704469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.704327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" event={"ID":"01845e69-4217-4e5e-b6b7-c255f3325067","Type":"ContainerDied","Data":"70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3"} Sep 10 19:39:05.704469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.704415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l" event={"ID":"01845e69-4217-4e5e-b6b7-c255f3325067","Type":"ContainerDied","Data":"b57038c4b4edf304864bf0e0a6c9c20c936304e05a9597a8a1dee893b44590a1"} Sep 10 19:39:05.704469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.704448 2576 scope.go:117] "RemoveContainer" containerID="70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3" Sep 10 19:39:05.715070 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.715050 2576 scope.go:117] "RemoveContainer" containerID="55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5" Sep 10 19:39:05.724138 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.724116 2576 scope.go:117] "RemoveContainer" containerID="70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3" Sep 10 19:39:05.724471 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:39:05.724449 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3\": container with ID starting with 70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3 not found: ID does not exist" containerID="70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3" Sep 10 19:39:05.724612 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.724483 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3"} err="failed to get container status \"70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3\": rpc error: code = NotFound desc = could not find container \"70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3\": container with ID starting with 70cbd0f454ec414203dd272f75e96ab8acec069de651d369142fb8492d2c18f3 not found: ID does not exist" Sep 10 19:39:05.724612 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.724511 2576 scope.go:117] "RemoveContainer" containerID="55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5" Sep 10 19:39:05.725119 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:39:05.725092 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5\": container with ID starting with 55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5 not found: ID does not exist" containerID="55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5" Sep 10 19:39:05.725237 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.725140 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5"} err="failed to get container status \"55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5\": rpc error: code = NotFound desc = could not find container \"55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5\": container with ID starting with 55a0cf272b8116f0187186e383ab046d5e41d9b47e13e93733633b63d48d9ff5 not found: ID does not exist" Sep 10 19:39:05.727825 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.727804 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l"] Sep 10 19:39:05.732092 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:05.732068 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b64b1c-openshift-default-7d5f8ff98d-th86l"] Sep 10 19:39:06.602080 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:06.602049 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" path="/var/lib/kubelet/pods/01845e69-4217-4e5e-b6b7-c255f3325067/volumes" Sep 10 19:39:10.702134 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.702096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:10.732067 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.732033 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8"] Sep 10 19:39:10.732558 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.732248 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" podUID="ad5356b9-f215-4857-a314-d4c0fb0edc69" containerName="payload-processing" containerID="cri-o://22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde" gracePeriod=30 Sep 10 19:39:10.741677 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.741649 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj"] Sep 10 19:39:10.741950 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.741917 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" podUID="c83610d2-37ff-45c1-9d84-3d2603461afd" containerName="payload-pre-processing" containerID="cri-o://9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4" gracePeriod=30 Sep 10 19:39:10.806468 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.806427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:10.906565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.906426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:10.933822 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.933793 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt"] Sep 10 19:39:10.935989 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:10.935967 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f15c09-s6jgt"] Sep 10 19:39:11.005248 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.005225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:11.022229 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.022201 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:39:11.025015 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.024990 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:39:11.119009 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.118971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83610d2-37ff-45c1-9d84-3d2603461afd-plugins-config-volume\") pod \"c83610d2-37ff-45c1-9d84-3d2603461afd\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " Sep 10 19:39:11.119186 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.119034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8nbl\" (UniqueName: \"kubernetes.io/projected/ad5356b9-f215-4857-a314-d4c0fb0edc69-kube-api-access-d8nbl\") pod \"ad5356b9-f215-4857-a314-d4c0fb0edc69\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " Sep 10 19:39:11.119186 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.119085 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5356b9-f215-4857-a314-d4c0fb0edc69-plugins-config-volume\") pod \"ad5356b9-f215-4857-a314-d4c0fb0edc69\" (UID: \"ad5356b9-f215-4857-a314-d4c0fb0edc69\") " Sep 10 19:39:11.119186 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.119162 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz9v2\" (UniqueName: \"kubernetes.io/projected/c83610d2-37ff-45c1-9d84-3d2603461afd-kube-api-access-rz9v2\") pod \"c83610d2-37ff-45c1-9d84-3d2603461afd\" (UID: \"c83610d2-37ff-45c1-9d84-3d2603461afd\") " Sep 10 19:39:11.119499 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.119465 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83610d2-37ff-45c1-9d84-3d2603461afd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c83610d2-37ff-45c1-9d84-3d2603461afd" (UID: "c83610d2-37ff-45c1-9d84-3d2603461afd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:11.119499 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.119471 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5356b9-f215-4857-a314-d4c0fb0edc69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad5356b9-f215-4857-a314-d4c0fb0edc69" (UID: "ad5356b9-f215-4857-a314-d4c0fb0edc69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:11.121469 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.121446 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5356b9-f215-4857-a314-d4c0fb0edc69-kube-api-access-d8nbl" (OuterVolumeSpecName: "kube-api-access-d8nbl") pod "ad5356b9-f215-4857-a314-d4c0fb0edc69" (UID: "ad5356b9-f215-4857-a314-d4c0fb0edc69"). InnerVolumeSpecName "kube-api-access-d8nbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:11.121617 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.121602 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83610d2-37ff-45c1-9d84-3d2603461afd-kube-api-access-rz9v2" (OuterVolumeSpecName: "kube-api-access-rz9v2") pod "c83610d2-37ff-45c1-9d84-3d2603461afd" (UID: "c83610d2-37ff-45c1-9d84-3d2603461afd"). InnerVolumeSpecName "kube-api-access-rz9v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:11.220833 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.220787 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83610d2-37ff-45c1-9d84-3d2603461afd-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:11.220833 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.220825 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8nbl\" (UniqueName: \"kubernetes.io/projected/ad5356b9-f215-4857-a314-d4c0fb0edc69-kube-api-access-d8nbl\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:11.220833 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.220838 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5356b9-f215-4857-a314-d4c0fb0edc69-plugins-config-volume\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:11.221085 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.220851 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9v2\" (UniqueName: \"kubernetes.io/projected/c83610d2-37ff-45c1-9d84-3d2603461afd-kube-api-access-rz9v2\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:11.317948 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.317915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:11.745180 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.745089 2576 generic.go:356] "Generic (PLEG): container finished" podID="c83610d2-37ff-45c1-9d84-3d2603461afd" containerID="9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4" exitCode=0 Sep 10 19:39:11.745180 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.745141 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" Sep 10 19:39:11.745773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.745179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" event={"ID":"c83610d2-37ff-45c1-9d84-3d2603461afd","Type":"ContainerDied","Data":"9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4"} Sep 10 19:39:11.745773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.745244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj" event={"ID":"c83610d2-37ff-45c1-9d84-3d2603461afd","Type":"ContainerDied","Data":"683be2ec8e37d6accd40d7298abd6ba10176253647e746417adcd07684f54651"} Sep 10 19:39:11.745773 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.745278 2576 scope.go:117] "RemoveContainer" containerID="9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4" Sep 10 19:39:11.746656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.746638 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad5356b9-f215-4857-a314-d4c0fb0edc69" containerID="22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde" exitCode=0 Sep 10 19:39:11.746724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.746668 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" event={"ID":"ad5356b9-f215-4857-a314-d4c0fb0edc69","Type":"ContainerDied","Data":"22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde"} Sep 10 19:39:11.746724 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.746682 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" Sep 10 19:39:11.746836 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.746722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8" event={"ID":"ad5356b9-f215-4857-a314-d4c0fb0edc69","Type":"ContainerDied","Data":"abf3b1e8475eb1117af5f79751c23af6b1f28f8b6a9f8dbe28186c575b9be82d"} Sep 10 19:39:11.756249 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.756135 2576 scope.go:117] "RemoveContainer" containerID="9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4" Sep 10 19:39:11.756542 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:39:11.756515 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4\": container with ID starting with 9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4 not found: ID does not exist" containerID="9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4" Sep 10 19:39:11.756649 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.756552 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4"} err="failed to get container status \"9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4\": rpc error: code = NotFound desc = could not find container \"9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4\": container with ID starting with 9b2ce23168e823878e130a8454e5d0ee3c2b754e0d8d03a3365b64ae230beff4 not found: ID does not exist" Sep 10 19:39:11.756649 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.756573 2576 scope.go:117] "RemoveContainer" containerID="22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde" Sep 10 19:39:11.765197 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.765178 2576 scope.go:117] "RemoveContainer" containerID="22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde" Sep 10 19:39:11.765617 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:39:11.765596 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde\": container with ID starting with 22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde not found: ID does not exist" containerID="22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde" Sep 10 19:39:11.765660 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.765637 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde"} err="failed to get container status \"22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde\": rpc error: code = NotFound desc = could not find container \"22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde\": container with ID starting with 22f9b5f10cb0b0880dffaf4eaf1cfb4d1df67168249718f7fc581118218cddde not found: ID does not exist" Sep 10 19:39:11.769457 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.769420 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj"] Sep 10 19:39:11.775340 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.775318 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f15c09-549d97df54-zfbbj"] Sep 10 19:39:11.790080 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.790049 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8"] Sep 10 19:39:11.796780 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:11.796756 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f15c09-86db6f4b8f-wf5k8"] Sep 10 19:39:12.602700 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:12.602669 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2609db0-1be7-4015-9c3f-9aa9a838b9e8" path="/var/lib/kubelet/pods/a2609db0-1be7-4015-9c3f-9aa9a838b9e8/volumes" Sep 10 19:39:12.603038 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:12.603025 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5356b9-f215-4857-a314-d4c0fb0edc69" path="/var/lib/kubelet/pods/ad5356b9-f215-4857-a314-d4c0fb0edc69/volumes" Sep 10 19:39:12.603335 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:12.603323 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83610d2-37ff-45c1-9d84-3d2603461afd" path="/var/lib/kubelet/pods/c83610d2-37ff-45c1-9d84-3d2603461afd/volumes" Sep 10 19:39:17.449788 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:17.449748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:17.555858 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:17.555833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:17.722684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:17.722597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:39:20.252841 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.252808 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55cd595499-pvrml"] Sep 10 19:39:20.253354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.253004 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55cd595499-pvrml" podUID="e8d7b5ef-65de-4f58-92da-f609ca9daff5" containerName="manager" containerID="cri-o://0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f" gracePeriod=30 Sep 10 19:39:20.503153 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.503088 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:39:20.611478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.611451 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-metrics-certs\") pod \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " Sep 10 19:39:20.611656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.611509 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkbgn\" (UniqueName: \"kubernetes.io/projected/e8d7b5ef-65de-4f58-92da-f609ca9daff5-kube-api-access-kkbgn\") pod \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " Sep 10 19:39:20.611656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.611571 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-webhook-cert\") pod \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\" (UID: \"e8d7b5ef-65de-4f58-92da-f609ca9daff5\") " Sep 10 19:39:20.613813 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.613772 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e8d7b5ef-65de-4f58-92da-f609ca9daff5" (UID: "e8d7b5ef-65de-4f58-92da-f609ca9daff5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:39:20.613939 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.613816 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d7b5ef-65de-4f58-92da-f609ca9daff5-kube-api-access-kkbgn" (OuterVolumeSpecName: "kube-api-access-kkbgn") pod "e8d7b5ef-65de-4f58-92da-f609ca9daff5" (UID: "e8d7b5ef-65de-4f58-92da-f609ca9daff5"). InnerVolumeSpecName "kube-api-access-kkbgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:20.613939 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.613893 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8d7b5ef-65de-4f58-92da-f609ca9daff5" (UID: "e8d7b5ef-65de-4f58-92da-f609ca9daff5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:39:20.712909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.712873 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-metrics-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:20.712909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.712907 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkbgn\" (UniqueName: \"kubernetes.io/projected/e8d7b5ef-65de-4f58-92da-f609ca9daff5-kube-api-access-kkbgn\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:20.712909 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.712916 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8d7b5ef-65de-4f58-92da-f609ca9daff5-webhook-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:39:20.804547 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.804508 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8d7b5ef-65de-4f58-92da-f609ca9daff5" containerID="0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f" exitCode=0 Sep 10 19:39:20.804730 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.804567 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-pvrml" Sep 10 19:39:20.804730 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.804593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-pvrml" event={"ID":"e8d7b5ef-65de-4f58-92da-f609ca9daff5","Type":"ContainerDied","Data":"0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f"} Sep 10 19:39:20.804730 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.804691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-pvrml" event={"ID":"e8d7b5ef-65de-4f58-92da-f609ca9daff5","Type":"ContainerDied","Data":"35437ffb9c8174a819d38978b831795cc6380fb550968e685504b7493cda17fa"} Sep 10 19:39:20.804730 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.804719 2576 scope.go:117] "RemoveContainer" containerID="0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f" Sep 10 19:39:20.816004 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.815985 2576 scope.go:117] "RemoveContainer" containerID="0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f" Sep 10 19:39:20.816331 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:39:20.816305 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f\": container with ID starting with 0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f not found: ID does not exist" containerID="0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f" Sep 10 19:39:20.816474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.816341 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f"} err="failed to get container status \"0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f\": rpc error: code = NotFound desc = could not find container \"0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f\": container with ID starting with 0c0b9b996cf0730230fefdedf826605b2c8385578ef1ead103a42b5b934b0b6f not found: ID does not exist" Sep 10 19:39:20.827208 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.827180 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55cd595499-pvrml"] Sep 10 19:39:20.829442 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:20.829418 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55cd595499-pvrml"] Sep 10 19:39:22.602364 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:22.602332 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d7b5ef-65de-4f58-92da-f609ca9daff5" path="/var/lib/kubelet/pods/e8d7b5ef-65de-4f58-92da-f609ca9daff5/volumes" Sep 10 19:39:35.816064 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:35.816030 2576 scope.go:117] "RemoveContainer" containerID="060c1361d190ab9416c9f6c3e415945393a57987624103ea1630a93087c838db" Sep 10 19:39:35.824838 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:39:35.824813 2576 scope.go:117] "RemoveContainer" containerID="7554edede140ef4932bda4ec78050ff5dc241bed0a39e572cbbcc68b37c02d9e" Sep 10 19:40:03.130217 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:03.130181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:40:03.229639 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:03.229606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:40:21.623874 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.623837 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55cd595499-22kbw"] Sep 10 19:40:21.624281 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624261 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83610d2-37ff-45c1-9d84-3d2603461afd" containerName="payload-pre-processing" Sep 10 19:40:21.624281 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624274 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83610d2-37ff-45c1-9d84-3d2603461afd" containerName="payload-pre-processing" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624286 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5356b9-f215-4857-a314-d4c0fb0edc69" containerName="payload-processing" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624292 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5356b9-f215-4857-a314-d4c0fb0edc69" containerName="payload-processing" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624299 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d7b5ef-65de-4f58-92da-f609ca9daff5" containerName="manager" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624308 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d7b5ef-65de-4f58-92da-f609ca9daff5" containerName="manager" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624316 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624327 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624342 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2609db0-1be7-4015-9c3f-9aa9a838b9e8" containerName="revoke-keys" Sep 10 19:40:21.624354 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624348 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2609db0-1be7-4015-9c3f-9aa9a838b9e8" containerName="revoke-keys" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624361 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624382 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624449 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2609db0-1be7-4015-9c3f-9aa9a838b9e8" containerName="revoke-keys" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624457 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624463 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5356b9-f215-4857-a314-d4c0fb0edc69" containerName="payload-processing" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624470 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d7b5ef-65de-4f58-92da-f609ca9daff5" containerName="manager" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624479 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01845e69-4217-4e5e-b6b7-c255f3325067" containerName="istio-proxy" Sep 10 19:40:21.624629 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.624490 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83610d2-37ff-45c1-9d84-3d2603461afd" containerName="payload-pre-processing" Sep 10 19:40:21.627790 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.627775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.631466 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.631443 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 19:40:21.631619 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.631494 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 19:40:21.631619 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.631517 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 19:40:21.631619 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.631571 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ltnwd\"" Sep 10 19:40:21.637774 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.637749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cd595499-22kbw"] Sep 10 19:40:21.774231 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.774191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-webhook-cert\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.774450 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.774253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-metrics-certs\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.774450 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.774327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgsj\" (UniqueName: \"kubernetes.io/projected/a13aaf59-00a2-4618-b07b-0082b7694bad-kube-api-access-jqgsj\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.875640 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.875544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgsj\" (UniqueName: \"kubernetes.io/projected/a13aaf59-00a2-4618-b07b-0082b7694bad-kube-api-access-jqgsj\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.875640 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.875625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-webhook-cert\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.875881 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.875671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-metrics-certs\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.878259 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.878223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-webhook-cert\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.878403 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.878260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-metrics-certs\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.884081 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.884061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgsj\" (UniqueName: \"kubernetes.io/projected/a13aaf59-00a2-4618-b07b-0082b7694bad-kube-api-access-jqgsj\") pod \"maas-controller-55cd595499-22kbw\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:21.939036 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:21.938993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:22.213397 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:22.213356 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cd595499-22kbw"] Sep 10 19:40:22.215305 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:40:22.215266 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13aaf59_00a2_4618_b07b_0082b7694bad.slice/crio-75d2c9217cfca56275cbd376f0cbecffd4a79369df622323b53a2679da0bffb7 WatchSource:0}: Error finding container 75d2c9217cfca56275cbd376f0cbecffd4a79369df622323b53a2679da0bffb7: Status 404 returned error can't find the container with id 75d2c9217cfca56275cbd376f0cbecffd4a79369df622323b53a2679da0bffb7 Sep 10 19:40:22.216580 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:22.216562 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:40:23.163923 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:23.163885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-22kbw" event={"ID":"a13aaf59-00a2-4618-b07b-0082b7694bad","Type":"ContainerStarted","Data":"d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c"} Sep 10 19:40:23.164333 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:23.163963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-22kbw" event={"ID":"a13aaf59-00a2-4618-b07b-0082b7694bad","Type":"ContainerStarted","Data":"75d2c9217cfca56275cbd376f0cbecffd4a79369df622323b53a2679da0bffb7"} Sep 10 19:40:23.164333 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:23.163984 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:23.183616 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:23.183558 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55cd595499-22kbw" podStartSLOduration=1.7146961699999999 podStartE2EDuration="2.183542438s" totalImagesPullingTime="468.846268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:40:23.18085769 +0000 UTC m=+1789.097513720" watchObservedRunningTime="2026-09-10 19:40:23.183542438 +0000 UTC m=+1789.100198467" Sep 10 19:40:34.171146 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:34.171113 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:34.721239 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:34.721207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:40:34.733688 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:34.733663 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:40:35.870854 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:35.870823 2576 scope.go:117] "RemoveContainer" containerID="d62a9ffb0f1c11c76e44dcd88f497ed13d02a7505fb93c7ebba07db2af25a001" Sep 10 19:40:35.879805 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:35.879782 2576 scope.go:117] "RemoveContainer" containerID="f6f2f6a2b27cc0d7301c78f26fd8ef79b81d2e0429cc73d6bea2e4a0e9bd4058" Sep 10 19:40:35.889241 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:35.889221 2576 scope.go:117] "RemoveContainer" containerID="39e358e03c36ec9de676463e58c190b5b8b9843f0ca568f2c259dff5c942b0e6" Sep 10 19:40:43.917400 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:43.917350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:40:45.503910 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.503871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:40:45.603522 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.603488 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55cd595499-22kbw"] Sep 10 19:40:45.603721 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.603697 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55cd595499-22kbw" podUID="a13aaf59-00a2-4618-b07b-0082b7694bad" containerName="manager" containerID="cri-o://d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c" gracePeriod=30 Sep 10 19:40:45.847181 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.847152 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:45.887342 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.887308 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-metrics-certs\") pod \"a13aaf59-00a2-4618-b07b-0082b7694bad\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " Sep 10 19:40:45.887563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.887353 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-webhook-cert\") pod \"a13aaf59-00a2-4618-b07b-0082b7694bad\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " Sep 10 19:40:45.887563 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.887402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqgsj\" (UniqueName: \"kubernetes.io/projected/a13aaf59-00a2-4618-b07b-0082b7694bad-kube-api-access-jqgsj\") pod \"a13aaf59-00a2-4618-b07b-0082b7694bad\" (UID: \"a13aaf59-00a2-4618-b07b-0082b7694bad\") " Sep 10 19:40:45.889881 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.889845 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13aaf59-00a2-4618-b07b-0082b7694bad-kube-api-access-jqgsj" (OuterVolumeSpecName: "kube-api-access-jqgsj") pod "a13aaf59-00a2-4618-b07b-0082b7694bad" (UID: "a13aaf59-00a2-4618-b07b-0082b7694bad"). InnerVolumeSpecName "kube-api-access-jqgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:40:45.890121 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.890092 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a13aaf59-00a2-4618-b07b-0082b7694bad" (UID: "a13aaf59-00a2-4618-b07b-0082b7694bad"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:40:45.890225 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.890127 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a13aaf59-00a2-4618-b07b-0082b7694bad" (UID: "a13aaf59-00a2-4618-b07b-0082b7694bad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:40:45.988598 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.988563 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-metrics-certs\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:40:45.988598 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.988593 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a13aaf59-00a2-4618-b07b-0082b7694bad-webhook-cert\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:40:45.988598 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:45.988602 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgsj\" (UniqueName: \"kubernetes.io/projected/a13aaf59-00a2-4618-b07b-0082b7694bad-kube-api-access-jqgsj\") on node \"ip-10-0-143-43.ec2.internal\" DevicePath \"\"" Sep 10 19:40:46.297535 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.297496 2576 generic.go:356] "Generic (PLEG): container finished" podID="a13aaf59-00a2-4618-b07b-0082b7694bad" containerID="d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c" exitCode=0 Sep 10 19:40:46.297743 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.297554 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-22kbw" Sep 10 19:40:46.297743 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.297585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-22kbw" event={"ID":"a13aaf59-00a2-4618-b07b-0082b7694bad","Type":"ContainerDied","Data":"d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c"} Sep 10 19:40:46.297743 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.297644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-22kbw" event={"ID":"a13aaf59-00a2-4618-b07b-0082b7694bad","Type":"ContainerDied","Data":"75d2c9217cfca56275cbd376f0cbecffd4a79369df622323b53a2679da0bffb7"} Sep 10 19:40:46.297743 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.297668 2576 scope.go:117] "RemoveContainer" containerID="d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c" Sep 10 19:40:46.307811 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.307793 2576 scope.go:117] "RemoveContainer" containerID="d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c" Sep 10 19:40:46.308115 ip-10-0-143-43 kubenswrapper[2576]: E0910 19:40:46.308095 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c\": container with ID starting with d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c not found: ID does not exist" containerID="d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c" Sep 10 19:40:46.308160 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.308124 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c"} err="failed to get container status \"d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c\": rpc error: code = NotFound desc = could not find container \"d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c\": container with ID starting with d017c2ad68cd6e17345aae582077699fc12106f02273824e06de6a2bc452d96c not found: ID does not exist" Sep 10 19:40:46.323663 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.323635 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55cd595499-22kbw"] Sep 10 19:40:46.334319 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.334298 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55cd595499-22kbw"] Sep 10 19:40:46.601634 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:46.601548 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a13aaf59-00a2-4618-b07b-0082b7694bad" path="/var/lib/kubelet/pods/a13aaf59-00a2-4618-b07b-0082b7694bad/volumes" Sep 10 19:40:47.153078 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.153036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:40:47.211202 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.211171 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55cd595499-tr4xr"] Sep 10 19:40:47.211564 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.211551 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13aaf59-00a2-4618-b07b-0082b7694bad" containerName="manager" Sep 10 19:40:47.211617 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.211566 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13aaf59-00a2-4618-b07b-0082b7694bad" containerName="manager" Sep 10 19:40:47.211653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.211636 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13aaf59-00a2-4618-b07b-0082b7694bad" containerName="manager" Sep 10 19:40:47.216019 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.216003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.219205 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.219185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 19:40:47.219963 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.219931 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ltnwd\"" Sep 10 19:40:47.220097 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.219995 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 19:40:47.220217 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.220204 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 19:40:47.231113 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.231084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cd595499-tr4xr"] Sep 10 19:40:47.300815 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.300776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51faceac-5584-4716-a7ea-7803cb285771-webhook-cert\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.300997 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.300827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51faceac-5584-4716-a7ea-7803cb285771-metrics-certs\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.300997 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.300878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9jc\" (UniqueName: \"kubernetes.io/projected/51faceac-5584-4716-a7ea-7803cb285771-kube-api-access-pr9jc\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.401748 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.401712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9jc\" (UniqueName: \"kubernetes.io/projected/51faceac-5584-4716-a7ea-7803cb285771-kube-api-access-pr9jc\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.401929 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.401799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51faceac-5584-4716-a7ea-7803cb285771-webhook-cert\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.401929 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.401830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51faceac-5584-4716-a7ea-7803cb285771-metrics-certs\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.404544 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.404485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51faceac-5584-4716-a7ea-7803cb285771-metrics-certs\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.404647 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.404598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51faceac-5584-4716-a7ea-7803cb285771-webhook-cert\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.411611 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.411581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9jc\" (UniqueName: \"kubernetes.io/projected/51faceac-5584-4716-a7ea-7803cb285771-kube-api-access-pr9jc\") pod \"maas-controller-55cd595499-tr4xr\" (UID: \"51faceac-5584-4716-a7ea-7803cb285771\") " pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.527176 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.527142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:47.804643 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:47.804609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cd595499-tr4xr"] Sep 10 19:40:47.806404 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:40:47.806344 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51faceac_5584_4716_a7ea_7803cb285771.slice/crio-2b31a3eb4cd42729403d0bf27d0ada1ac6f468d1b9a74ed326a5d48716681b8a WatchSource:0}: Error finding container 2b31a3eb4cd42729403d0bf27d0ada1ac6f468d1b9a74ed326a5d48716681b8a: Status 404 returned error can't find the container with id 2b31a3eb4cd42729403d0bf27d0ada1ac6f468d1b9a74ed326a5d48716681b8a Sep 10 19:40:48.310903 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:48.310871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-tr4xr" event={"ID":"51faceac-5584-4716-a7ea-7803cb285771","Type":"ContainerStarted","Data":"2b31a3eb4cd42729403d0bf27d0ada1ac6f468d1b9a74ed326a5d48716681b8a"} Sep 10 19:40:49.321565 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:49.321522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cd595499-tr4xr" event={"ID":"51faceac-5584-4716-a7ea-7803cb285771","Type":"ContainerStarted","Data":"f3a0879c6402944d113d4a59fde41036a50263f26e281a1ad2cedd9db676b12e"} Sep 10 19:40:49.322034 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:49.321628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:40:49.342618 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:40:49.342567 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55cd595499-tr4xr" podStartSLOduration=1.895744792 podStartE2EDuration="2.342551373s" totalImagesPullingTime="446.806581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:40:49.341491047 +0000 UTC m=+1815.258147075" watchObservedRunningTime="2026-09-10 19:40:49.342551373 +0000 UTC m=+1815.259207399" Sep 10 19:41:00.331546 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:41:00.331502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55cd595499-tr4xr" Sep 10 19:41:06.947067 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:41:06.947031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:41:19.711574 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:41:19.711510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:41:20.970010 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:41:20.969965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:42:31.936125 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:42:31.936065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:43:35.937871 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:43:35.937791 2576 scope.go:117] "RemoveContainer" containerID="394a7e8e9c33ebbe25757501426508a58bf46a87ecfbcbb18191a9e16e0fc539" Sep 10 19:43:35.946653 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:43:35.946632 2576 scope.go:117] "RemoveContainer" containerID="d388059d9e48b37c9bacb40a3cf12b4edcaf0fb28d4f360ce79cd5d9881edfd4" Sep 10 19:43:52.474568 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:43:52.474528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:43:57.024578 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:43:57.024546 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:44:16.803083 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:44:16.803046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:44:18.495422 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:44:18.495387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:44:35.612694 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:44:35.612662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:44:35.970187 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:44:35.970108 2576 scope.go:117] "RemoveContainer" containerID="01ba9ea3c62eede22a99a04526b4ea21106cf83890b7d4b529179865b28aaa68" Sep 10 19:45:00.144948 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.144866 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb"] Sep 10 19:45:00.147842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.147824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" Sep 10 19:45:00.151551 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.151524 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\"" Sep 10 19:45:00.159197 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.159177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" Sep 10 19:45:00.166474 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.166450 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb"] Sep 10 19:45:00.443079 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.443003 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb"] Sep 10 19:45:00.446497 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:45:00.446470 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097bd531_056e_4472_8c34_c44a2cde07f9.slice/crio-e116aa10f510021a6edd013a83ea4418d8f5b40ffd22482d64ec4c0e39750c48 WatchSource:0}: Error finding container e116aa10f510021a6edd013a83ea4418d8f5b40ffd22482d64ec4c0e39750c48: Status 404 returned error can't find the container with id e116aa10f510021a6edd013a83ea4418d8f5b40ffd22482d64ec4c0e39750c48 Sep 10 19:45:00.721763 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.721660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" event={"ID":"097bd531-056e-4472-8c34-c44a2cde07f9","Type":"ContainerStarted","Data":"645b5672e61316a42f21af663622b79373b921fd7c30a50030906d7c69253c4b"} Sep 10 19:45:00.721763 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.721720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" event={"ID":"097bd531-056e-4472-8c34-c44a2cde07f9","Type":"ContainerStarted","Data":"e116aa10f510021a6edd013a83ea4418d8f5b40ffd22482d64ec4c0e39750c48"} Sep 10 19:45:00.739761 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:00.739713 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" podStartSLOduration=0.739695834 podStartE2EDuration="739.695834ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:00.737978478 +0000 UTC m=+2066.654634508" watchObservedRunningTime="2026-09-10 19:45:00.739695834 +0000 UTC m=+2066.656351863" Sep 10 19:45:01.729242 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:01.729153 2576 generic.go:356] "Generic (PLEG): container finished" podID="097bd531-056e-4472-8c34-c44a2cde07f9" containerID="645b5672e61316a42f21af663622b79373b921fd7c30a50030906d7c69253c4b" exitCode=0 Sep 10 19:45:01.729242 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:01.729191 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" event={"ID":"097bd531-056e-4472-8c34-c44a2cde07f9","Type":"ContainerDied","Data":"645b5672e61316a42f21af663622b79373b921fd7c30a50030906d7c69253c4b"} Sep 10 19:45:02.878126 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:02.878102 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" Sep 10 19:45:03.741428 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:03.741392 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" Sep 10 19:45:03.741428 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:03.741419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817825-8cpqb" event={"ID":"097bd531-056e-4472-8c34-c44a2cde07f9","Type":"ContainerDied","Data":"e116aa10f510021a6edd013a83ea4418d8f5b40ffd22482d64ec4c0e39750c48"} Sep 10 19:45:03.741644 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:03.741465 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e116aa10f510021a6edd013a83ea4418d8f5b40ffd22482d64ec4c0e39750c48" Sep 10 19:45:16.853243 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:16.853211 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cd595499-tr4xr_51faceac-5584-4716-a7ea-7803cb285771/manager/0.log" Sep 10 19:45:17.101480 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:17.101445 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fpn29_ae3bb302-cb2b-436c-9df6-09384999fc94/manager/0.log" Sep 10 19:45:17.342895 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:17.342868 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-gct5x_c2c03473-d443-4ee3-b9ad-18d41c71da27/copy-manifests/0.log" Sep 10 19:45:17.355022 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:17.354990 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-gct5x_c2c03473-d443-4ee3-b9ad-18d41c71da27/manager/0.log" Sep 10 19:45:18.029844 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:18.029812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817810-6g2b9_a0070b21-bb9c-42b2-90c6-3001087a798a/cleanup/0.log" Sep 10 19:45:18.140856 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:18.140824 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817825-8cpqb_097bd531-056e-4472-8c34-c44a2cde07f9/cleanup/0.log" Sep 10 19:45:20.056995 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:20.056965 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cjlf52_e41275c9-a0cd-4f3c-b487-4510c5644384/istio-proxy/0.log" Sep 10 19:45:20.168258 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:20.168223 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lswj9_06dc50da-ff3e-4041-a38d-051cc83baf6f/discovery/0.log" Sep 10 19:45:20.625684 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:20.625653 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g92sd_59022858-10e9-425b-9c9a-dc023040fe85/payload-pre-processing/0.log" Sep 10 19:45:20.738607 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:20.738577 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fbqwq_78633393-0b05-45f7-925c-2c41aceca7b4/payload-processing/0.log" Sep 10 19:45:20.852064 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:20.852032 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-588979fc78-gdjfk_e01c9a3d-8c0f-482a-82fd-9e6d416f4647/router/0.log" Sep 10 19:45:21.413183 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:21.413155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-c2gzf_8cec2d25-d6ba-4b85-8c37-967156c18598/main/0.log" Sep 10 19:45:21.525994 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:21.525961 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4vccm_d6887099-3834-46d4-81c5-d8e83ab7e4bc/main/0.log" Sep 10 19:45:27.029154 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:27.029121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:45:27.117810 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:27.117777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" limit=1024 Sep 10 19:45:30.145839 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:30.145806 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qxlb7_b14c0d55-d660-472c-ad93-f42bf3893fe9/global-pull-secret-syncer/0.log" Sep 10 19:45:30.202235 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:30.202204 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f6t55_33146da8-d5b7-4b63-aebf-2c12ae94fb5a/konnectivity-agent/0.log" Sep 10 19:45:30.357529 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:30.357497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-43.ec2.internal_eec5dd9f88d629eb8d902b14a241ffc9/haproxy/0.log" Sep 10 19:45:34.758951 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:34.758872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:45:34.771486 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:34.771462 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:45:35.993182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:35.993143 2576 scope.go:117] "RemoveContainer" containerID="e069b217dc42c2a82d192dea32753c6e1e9aebab2eb020c53ee962a2b2caa24c" Sep 10 19:45:36.004707 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:36.004681 2576 scope.go:117] "RemoveContainer" containerID="7963fadbcfa3d975a638cac99c9943e17f02a3eadf4c57be7e1ec7956c25dc97" Sep 10 19:45:37.037097 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:37.037066 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-pnvfl_8d4d0730-72f3-4a8f-b169-b008b59ad4b0/cluster-monitoring-operator/0.log" Sep 10 19:45:37.212768 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:37.212744 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7zj7q_fcba22d5-19ed-435d-8f74-dad0a96395c0/node-exporter/0.log" Sep 10 19:45:37.237317 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:37.237289 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7zj7q_fcba22d5-19ed-435d-8f74-dad0a96395c0/kube-rbac-proxy/0.log" Sep 10 19:45:37.265149 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:37.265113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7zj7q_fcba22d5-19ed-435d-8f74-dad0a96395c0/init-textfile/0.log" Sep 10 19:45:37.972472 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:37.972443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-8npks_9c3342ec-124f-40a3-b8f9-9b4d945e380d/prometheus-operator-admission-webhook/0.log" Sep 10 19:45:40.232309 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.232275 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/2.log" Sep 10 19:45:40.236730 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.236709 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lq654_264d6101-b27d-4c83-bfe3-f70d9dcb0f74/console-operator/3.log" Sep 10 19:45:40.578182 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.578150 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l"] Sep 10 19:45:40.578534 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.578520 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097bd531-056e-4472-8c34-c44a2cde07f9" containerName="cleanup" Sep 10 19:45:40.578586 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.578535 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="097bd531-056e-4472-8c34-c44a2cde07f9" containerName="cleanup" Sep 10 19:45:40.578667 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.578596 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="097bd531-056e-4472-8c34-c44a2cde07f9" containerName="cleanup" Sep 10 19:45:40.581610 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.581593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.584185 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.584165 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vr76\"/\"kube-root-ca.crt\"" Sep 10 19:45:40.585115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.585092 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vr76\"/\"default-dockercfg-j88m6\"" Sep 10 19:45:40.585115 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.585110 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vr76\"/\"openshift-service-ca.crt\"" Sep 10 19:45:40.592431 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.592406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l"] Sep 10 19:45:40.690705 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.690659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-proc\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.690705 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.690708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-lib-modules\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.690926 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.690729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-sys\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.690926 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.690816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-podres\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.690926 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.690849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2pn\" (UniqueName: \"kubernetes.io/projected/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-kube-api-access-vg2pn\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792021 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.791988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-podres\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792196 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2pn\" (UniqueName: \"kubernetes.io/projected/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-kube-api-access-vg2pn\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792196 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-proc\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792276 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-proc\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792276 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-podres\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792276 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-lib-modules\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792412 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-sys\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792412 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-lib-modules\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.792412 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.792344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-sys\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.802285 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.802259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2pn\" (UniqueName: \"kubernetes.io/projected/25c9af46-6896-4d7b-b3a2-d7cfb2019d86-kube-api-access-vg2pn\") pod \"perf-node-gather-daemonset-zlp2l\" (UID: \"25c9af46-6896-4d7b-b3a2-d7cfb2019d86\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:40.893003 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:40.892915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:41.175068 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:41.175044 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l"] Sep 10 19:45:41.176612 ip-10-0-143-43 kubenswrapper[2576]: W0910 19:45:41.176576 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod25c9af46_6896_4d7b_b3a2_d7cfb2019d86.slice/crio-eab42af9cc5d1c296c2719f49d26224b20e88df76ab15ba15665e869c5dc2a43 WatchSource:0}: Error finding container eab42af9cc5d1c296c2719f49d26224b20e88df76ab15ba15665e869c5dc2a43: Status 404 returned error can't find the container with id eab42af9cc5d1c296c2719f49d26224b20e88df76ab15ba15665e869c5dc2a43 Sep 10 19:45:41.642150 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:41.642120 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-zxqxr_e5516b97-0f82-4a19-86a6-ce51e793fddb/volume-data-source-validator/0.log" Sep 10 19:45:41.946654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:41.946563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" event={"ID":"25c9af46-6896-4d7b-b3a2-d7cfb2019d86","Type":"ContainerStarted","Data":"d002c89e1175931b0e8b08a584e1d681aee44a1ecbe449313e9c953a9fa78ecd"} Sep 10 19:45:41.946654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:41.946631 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:41.946654 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:41.946653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" event={"ID":"25c9af46-6896-4d7b-b3a2-d7cfb2019d86","Type":"ContainerStarted","Data":"eab42af9cc5d1c296c2719f49d26224b20e88df76ab15ba15665e869c5dc2a43"} Sep 10 19:45:41.969208 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:41.969162 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" podStartSLOduration=1.969145298 podStartE2EDuration="1.969145298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:41.968988601 +0000 UTC m=+2107.885644630" watchObservedRunningTime="2026-09-10 19:45:41.969145298 +0000 UTC m=+2107.885801326" Sep 10 19:45:42.805823 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:42.805793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t7gm6_d9ff1c36-92f9-4f39-82da-95672fcd5de9/dns/0.log" Sep 10 19:45:42.829800 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:42.829771 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t7gm6_d9ff1c36-92f9-4f39-82da-95672fcd5de9/kube-rbac-proxy/0.log" Sep 10 19:45:42.859273 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:42.859247 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6qjq4_0f6c2cf3-35b5-4054-8b11-58f48bc41faf/dns-node-resolver/0.log" Sep 10 19:45:43.551136 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:43.551108 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2wk66_e94d7c0b-9cbd-4e83-ab81-b24c51c1b0bd/node-ca/0.log" Sep 10 19:45:44.613577 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:44.613550 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cjlf52_e41275c9-a0cd-4f3c-b487-4510c5644384/istio-proxy/0.log" Sep 10 19:45:44.839970 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:44.839936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lswj9_06dc50da-ff3e-4041-a38d-051cc83baf6f/discovery/0.log" Sep 10 19:45:44.971528 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:44.971449 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g92sd_59022858-10e9-425b-9c9a-dc023040fe85/payload-pre-processing/0.log" Sep 10 19:45:45.006153 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:45.006120 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fbqwq_78633393-0b05-45f7-925c-2c41aceca7b4/payload-processing/0.log" Sep 10 19:45:45.030842 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:45.030815 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-588979fc78-gdjfk_e01c9a3d-8c0f-482a-82fd-9e6d416f4647/router/0.log" Sep 10 19:45:45.990849 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:45.990821 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v4j2x_124cc65e-d0ed-4659-99d9-cb8a2c7a8763/serve-healthcheck-canary/0.log" Sep 10 19:45:46.484235 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:46.484203 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-slhxt_ec9ceda2-9fc5-439b-86c2-171a6effa80d/insights-operator/0.log" Sep 10 19:45:46.484446 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:46.484414 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-slhxt_ec9ceda2-9fc5-439b-86c2-171a6effa80d/insights-operator/1.log" Sep 10 19:45:46.655904 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:46.655863 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whqpq_996a8504-cb25-471d-a86e-f34bd0f9915f/kube-rbac-proxy/0.log" Sep 10 19:45:46.678843 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:46.678817 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whqpq_996a8504-cb25-471d-a86e-f34bd0f9915f/exporter/0.log" Sep 10 19:45:46.706028 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:46.706000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whqpq_996a8504-cb25-471d-a86e-f34bd0f9915f/extractor/0.log" Sep 10 19:45:47.962682 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:47.962651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-zlp2l" Sep 10 19:45:49.413948 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:49.413917 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cd595499-tr4xr_51faceac-5584-4716-a7ea-7803cb285771/manager/0.log" Sep 10 19:45:49.532315 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:49.532237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fpn29_ae3bb302-cb2b-436c-9df6-09384999fc94/manager/0.log" Sep 10 19:45:49.672347 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:49.672314 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-gct5x_c2c03473-d443-4ee3-b9ad-18d41c71da27/manager/0.log" Sep 10 19:45:49.692567 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:49.692542 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-gct5x_c2c03473-d443-4ee3-b9ad-18d41c71da27/copy-manifests/0.log" Sep 10 19:45:50.770553 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:50.770525 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cjlf52_e41275c9-a0cd-4f3c-b487-4510c5644384/istio-proxy/0.log" Sep 10 19:45:50.999784 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:50.999751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lswj9_06dc50da-ff3e-4041-a38d-051cc83baf6f/discovery/0.log" Sep 10 19:45:51.101831 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:51.101803 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g92sd_59022858-10e9-425b-9c9a-dc023040fe85/payload-pre-processing/0.log" Sep 10 19:45:51.126344 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:51.126312 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fbqwq_78633393-0b05-45f7-925c-2c41aceca7b4/payload-processing/0.log" Sep 10 19:45:51.147656 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:51.147627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-588979fc78-gdjfk_e01c9a3d-8c0f-482a-82fd-9e6d416f4647/router/0.log" Sep 10 19:45:51.348024 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:51.347992 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6748c6bd4b-5q9jz_09d027f9-967d-45bc-8770-303097899376/manager/0.log" Sep 10 19:45:51.418261 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:51.418234 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5hq4r_3fdb8b2a-1ac1-4a02-b9cb-eb26955fbb47/openshift-lws-operator/0.log" Sep 10 19:45:52.674696 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:52.674670 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cjlf52_e41275c9-a0cd-4f3c-b487-4510c5644384/istio-proxy/0.log" Sep 10 19:45:52.903519 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:52.903490 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lswj9_06dc50da-ff3e-4041-a38d-051cc83baf6f/discovery/0.log" Sep 10 19:45:53.000358 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:53.000329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g92sd_59022858-10e9-425b-9c9a-dc023040fe85/payload-pre-processing/0.log" Sep 10 19:45:53.027768 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:53.027738 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fbqwq_78633393-0b05-45f7-925c-2c41aceca7b4/payload-processing/0.log" Sep 10 19:45:53.043004 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:53.042981 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-588979fc78-gdjfk_e01c9a3d-8c0f-482a-82fd-9e6d416f4647/router/0.log" Sep 10 19:45:56.943145 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:56.943072 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-pwbxn_226486d2-fe60-4143-9151-43c5cae901ee/kube-storage-version-migrator-operator/1.log" Sep 10 19:45:56.943947 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:56.943929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-pwbxn_226486d2-fe60-4143-9151-43c5cae901ee/kube-storage-version-migrator-operator/0.log" Sep 10 19:45:58.299525 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:58.299497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dnd8d_7d3d0fc0-11c5-4f7e-add8-707f5640cb28/kube-multus-additional-cni-plugins/0.log" Sep 10 19:45:58.324478 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:58.324451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dnd8d_7d3d0fc0-11c5-4f7e-add8-707f5640cb28/egress-router-binary-copy/0.log" Sep 10 19:45:58.349703 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:58.349676 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dnd8d_7d3d0fc0-11c5-4f7e-add8-707f5640cb28/cni-plugins/0.log" Sep 10 19:45:58.380444 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:58.380416 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dnd8d_7d3d0fc0-11c5-4f7e-add8-707f5640cb28/bond-cni-plugin/0.log" Sep 10 19:45:58.407212 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:58.407183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dnd8d_7d3d0fc0-11c5-4f7e-add8-707f5640cb28/routeoverride-cni/0.log" Sep 10 19:45:58.437955 ip-10-0-143-43 kubenswrapper[2576]: I0910 19:45:58.437914 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dnd8d_7d3d0fc0-11c5-4f7e-add8-707f5640cb28/whereabouts-cni-bincopy/0.log"